What are the ways I can make my circuit communicate with a computer?
There are quite a few different ways you could achieve this. Here are a few:
You could program a small microcontroller like a Microchip PIC16F84A to watch the button and the LED. Whenever the button is pressed, the PIC16F84A sends a character through a FTDI serial chip to a USB port on your computer. On your computer, a program written in Processing watches the USB port and updates the pictures on the screen. Total cost: $20 for some chips and a breadboard.
You could buy a Bluetooth Arduino with a prototyping shield. The Arduino talks over a Bluetooth chip to a Bluetooth card in your PC. A program written in Python listens to the Bluetooth card and draws the appropriate picture on the screen with the Pygame library. Total cost: $150 for a Bluetooth Arduino.
You could attach a webcam to your PC that watches your LED and the button. A program written in Ruby is analyzing every image coming from the webcam, comparing it to the previous image. When it detects a change in the region of the image near the button, it looks at the color of the pixels in the LED region and updates a value in a database. A webpage auto-refreshes using the jQuery timer plugin, and updates an image on the screen based on the value in the database. Total cost: $25 for a crappy webcam.
You could find an old PC with a parallel port and install Ubuntu Linux on it. You connect the LED and the button to pins on the parallel port and then write a program in C that reads address 0x378. Based on the data returned, it makes calls into a C graphics library that does the appropriate screen drawing. Total cost: free, if you can find a PC old enough.
You replace the power button on your computer with the button in question, and the power LED with the LED in question. You replace your BIOS with a version of OpenBIOS that has been modified to display a green circle on the screen, and never boot any operating system. Total cost: probably the cost of a new PC.
But in all seriousness
I think I'd recommend an Arduino and Processing. They're simple and great for beginners. Total cost: $30 for the Arduino.
Best of luck.
Go get the book "Making Things Talk" by Tom Igoe: http://oreilly.com/catalog/9780596510510
It's not about speech, it's about all the different methods of making your different devices-- e.g., your computer and your circuit --talk to each other. In it, you'll find the gory details of all the methods people are outlining in the other answers here, complete with circuit diagrams and source code (where appropriate).
If you'd like to learn any or all the different ways to do this, you couldn't have a better starting point.
Python + FT245
Drop the PyUSB module, and suddenly it takes only three lines (well, four, if you include the import) of code to get Python talking to meatspace through 8 Parallel IO lines.
import d2xx
hardwareHandle = d2xx.open(0)
#Open the First FTDI device on the computer
hardwareHandle.setBitMode(0xff, 0x01)
#Put the hardware in Async Bit-bang mode, set all pins as outputs
#Finally
hardwareHandle.write(data)
#And suddenly
#`data` shows up on the FT245 pins!
hardwareHandle.setBitMode(0x00, 0x01)
#set the IO lines to inputs (the first byte is the IO mask)
input = hardwareHandle.read()
#and input is equal to the value on the IO lines!
It's dead-simple parallel IO from a high-level language.
As a bonus, the hardware costs a total of $17.95