As to testing with the multimeter. Not knowing what your ability with one is, but you say you've only recently purchased one, let's start with some simple stuff. After installing and connecting the battery in the car select a suitable voltage setting on your meter. This will Depend on the type of meter you have. My older meter is a very common cheaper one where the voltage has to be selected manually whereas my newer, but still not ultra expensive one, is autoranging so senses what the battery voltage is and displays it appropriately.
Here's a picture of them both in the switched off position:
The green one on the left is the autoranging one and the yellow one on the right is my "good old buddy". The next picture shows them with appropriate settings selected for connection to a car battery:
The green one just needs it's selector turned to the DC Voltage position - Most meters indicate DC with the straight and broken line symbol you can see and AC Voltage with a wavy (sine wave) line symbol, which again you can see on the meter selector - The yellow one has had it's selector turned to the 20 volt position. You need to select a range greater than the voltage you are expecting to measure but not too much or you won't get much refinement in the reading. (if in doubt start with a very high scale and work down as the voltage is revealed).
Now put the probes on the battery terminals. Positive (red) to battery pos and negative (black) to battery neg. Digital meters are very forgiving - usually - so if you connect the "wrong" way round all that'll happen is you'll see a negative sign in front of the numbers on the screen. Put the probes directly on the actual battery posts for these first readings so you are obtaining direct readings from the battery itself without the possibility of poor battery terminal connections confusing the issue.
Now look at what the reading is. In theory each cell of a lead acid battery produces 2.2 volts when fully charged so a typical "12 volt" battery, with it's six cells, is more accurately a 13 volt battery in fact. In practice a 12 volt car battery is generally considered to be "fully charged" when showing around 12.6 or more volts. I've just dodged out to my garage and checked the big battery I keep for jump starting and it's showing 12.66 volts. I know it will show 12.9 volts when fully charged so I'll be sticking it on charge for a wee while tonight. Interestingly, once an open circuit battery (so a battery which is not connected to anything and passing no current) is considered "dead" when showing less than 12 volts. However people get very confused by this so a better way of checking your battery's performance is to keep the meter probes on the battery posts and operate the starter motor. This puts a heavy demand on the battery and you're looking for a minimum reading of around 10 volts (or slightly under).
It's perfectly possible to get a good voltage reading, say 12.5 volts or more, when the battery is not under load but a poor reading, say under 9 volts, when cranking due to internal problems in the battery. This is why i say to put the probes directly on the battery posts because if you have them on the clamps you'll get a poor reading if the battery is poor but also if the terminal clamps are not making good contact with their battery posts (and it only takes one) then you'll also get a poor voltage showing and you won't know that actually the battery may be Ok and all you need to do is clean up the posts/clamps to get a good contact!
So, if that all works out Ok. So good open circuit voltage (around 12.5 volts or more, of course it may be less just because the battery needs charging so don't condemn the battery until you've charged it and tried the test again) and more than 10 volts, or close too, under cranking, the next thing to check is what voltage is registered with the engine running.
So keeping the probes on the battery posts start the engine and see what the voltage is. It should jump up immediately to around 13.5 to 14.5 volts. This is because the alternator output is roughly a volt or so above the dormant battery voltage so that it can force electricity through the battery. With modern alternators you would expect to find this charging voltage to be there right from tickover up to max revs although you may find with some that there will be a slight increase in voltage as the revs lift away from tickover. If you're seeing the higher voltage and it's holding steady by the time you hit 1500 rpm there's probably not much wrong and the alternator is charging the battery.
The above is very much just simple checking and there's a lot more you can check for but it's a good starting point. Sticking to checking voltages is a pretty safe thing to do as the meter has very high internal resistance when taking voltage readings so shorting things out is quite difficult to do. Beware of selecting current (amperage) or resistance (ohms) scales where you can do lots of damage if you stick the probes in the wrong place.