By saying you want to 'remove the background', I'm assuming you want to have a transparent background. If the current background is pretty much a solid color (black or white, whatever the lid of your scanner is), then the easiest way may just be to use the "magic wand" tool. Click it anywhere on the 'background' of the layer and it should become 'selected' (with the marching ants). Now just tap the 'delete' key lightly (if you hit it too hard the keyboard may complain!) and the 'background should disappear. If, instead, it bacomes another color, it may be that you have not converted the image into 'RGB'. Back up one step, make the conversion and see if that works.
Be aware, also, that there will always be a background unless you save the image as a GIF, PNG-32(?), PSD, or maybe TIFF. If displaying on the web, GIF may be the easiest, but PNG will work when supported by the browser. If you are importing the image into a page layout program, it might be better to use a TIFF (with at least 150dpi resolution).
Of course, I don't have any experience with the 'real' PS, but the above works great in Elements!
BTW, to make the elipse a cirle (or the rectangle stay a square) try holding the shift key while dragging the mouse.
Jim C.