How Port Scanner Work

Port Scanning Basic Techniques
The simplest port scan tries (i.e., sends a carefully constructed packet with a chosen destination port number) each of the ports from 0 to 65535 on the victim to see which ones are open.
TCP connect():- The connect() system call provided by an OS is used to open a connection to every interesting port on the machine. If the port is listening, connect() will succeed, otherwise the port isn't reachable.
Strobe -A strobe does a narrower scan, only looking for those services the attacker knows how to exploit. The name comes from one of the original TCP scanning programs, though now virtually all scanning tools include this feature.
The ident protocol allows for the disclosure of the username of the owner of any process connected via TCP, even if that process didn't initiate the connection. So, e.g., one can connect to port 80 and then use identd to find out whether the HTTP server is running as root.
Port Scanning Advanced Techniques
One problem, from the perspective of the attacker attempting to scan a port, is that services listening on these ports log scans. They see an incoming connection, but no data, so an error is logged. There exist a number of stealth scan techniques to avoid this. A stealth scan is a kind of scan that is designed to go undetected by auditing tools. Obviously, this is a race between the hacker and firewall vendors - what are considered stealth scans now may not be so in a few months once the firewall vendor becomes aware of such techniques.
Port scanners scan a host rapidly by firing off packets at different ports. So, scanning very slowly (taking a day or more) becomes a stealth technique. Another stealth scanning technique is "inverse mapping", where you try to find out all hosts on a network by generating "host unreachable" ICMP-messages for those IPs that do not exist. Since these messages may be generated by any TCP/IP packet one may send meaningless packets (e.g. RST packets sent without any previous packet).
Fragmented packet Port Scan
The scanner splits the TCP header into several IP fragments. This bypasses some packet filter firewalls because they cannot see a complete TCP header that can match their filter rules. Some packet filters and firewalls do queue all IP fragments, but many networks cannot afford the performance loss caused by the queuing.
SYN scan
This technique is also called half-open scanning, because a TCP connection is not completed. A SYN packet is sent (as if we are going to open a connection), and the target host responds with a SYN+ACK, this indicates the port is listening, and an RST indicates a non- listener. The server process is never informed by the TCP layer because the connection did not complete.
FIN scan
The typical TCP scan attempts to open connections (at least part way). Another technique sends erroneous packets at a port, expecting that open listening ports will send back different error messages than closed ports. The scanner sends a FIN packet, which should close a connection that is open. Closed ports reply to a FIN packet with a RST. Open ports, on the other hand, ignore the packet in question. This is required TCP behavior.
If no service is listening at the target port, the operating system will generate an error message. If a service is listening, the operating system will silently drop the incoming packet. Therefore, silence indicates the presence of a service at the port. However, since packets can be dropped accidentally on the wire or blocked by firewalls, this isn't a very effective scan.
Other techniques that have been used consist of XMAS scans where all flags in the TCP packet are set, or NULL scans where none of the bits are set. However, different operating systems respond differently to these scans, and it becomes important to identify the OS and even its version and patch level.
Bounce Scan
The ability to hide their tracks is important to attackers. Therefore, attackers scour the Internet looking for systems they can bounce their attacks through.
FTP bounce scanning takes advantage of a vulnerability of the FTP protocol itself. It requires support for proxy ftp connections. This bouncing through an FTP server hides where the attacker comes from. This technique is similar to IP spoofing in that it hides where the attacker comes from. For example, badwebsitexyz.com establishes a control connection to the FTP server-PI (protocol interpreter) of say, badspiderbites.com, then requests that the server-PI initiate an active server-DTP (data transfer process) to send a file anywhere on the Internet.
A port scanner can exploit this to scan TCP ports from a proxy ftp server. Thus you could connect to an FTP server behind a firewall, and then scan ports that are more likely to be blocked (e.g., port 139). If the ftp server allows reading from and writing to a directory (such as /incoming), you can send arbitrary data to ports that you do find open.
The advantages to this approach are obvious (harder to trace, potential to bypass firewalls). The main disadvantages are that it is slow, and that many FTP server implementations have finally disabled the proxy "feature".
Finger
Most finger servers allow commands to be forwarded through them. Finger supports recursive queries. A query such as "rob@foo@bar" will ask "bar" to resolve "rob@foo", causing "bar" to query "foo". This technique can be used to hide the original source of the request.
E-mail: Spammers try to relay their spam through SMTP servers. As a result, probes for SMTP are commonly seen by machines on the Internet.
SOCKS Allows almost any protocol to be tunneled through the intermediate machine. As a result, attackers probing for SOCKS is common scan seen on the Internet.
HTTP proxy: Most web servers support proxying so that all web traffic can be directed to a single server for filtering as well as caching to improve performance. A lot of these servers are misconfigured to allow proxying of any request from the Internet, allowing attackers to relay attacks against web sites through a third party. Probes for HTTP proxies are one of the more common scans seen today.
IRC BNC: Attackers love to hide their IRC identities by bouncing their connections through other machines. A particular program called "BNC" is used for this purpose on compromised machines.
UDP Scanning
Port scanning usually means scanning for TCP ports, which are connection-oriented and therefore give good feedback to the attacker. UDP responds in a different manner. In order to find UDP ports, the attacker generally sends empty UDP datagrams. If the port is listening, the service should send back an error message or ignore the incoming datagram. If the port is closed, then most operating systems send back an "ICMP Port Unreachable" message. Thus, you can find out if a port is NOT open, and by exclusion determine which ports are open. Neither UDP packets, nor the ICMP errors are guaranteed to arrive, so UDP scanners of this sort must also implement retransmission of packets that appear to be lost (or you will get a bunch of false positives).
Also, this scanning technique is slow because of compensation for machines that implement the suggestions of RFC 1812 and limit ICMP error message rate. For example, a kernal may limit destination unreachable message generation to 80 per 4 seconds, with a 1/4 second penalty if that is exceeded.
Some people think UDP scanning is pointless - not so. Sometimes for example, Rpcbind can be found hiding on an undocumented UDP port somewhere above 32770. So it doesn't matter that port 111 is blocked by the firewall. But can you find which of the more than 30,000 high ports it is listening on? With a UDP scanner you can.
ICMP Scan
This isn't really port scanning, since ICMP does not have a port abstraction. But it is sometimes useful to determine what hosts in a network are up by pinging them all. ICMP scanning can be done in parallel, so it can be quite fast.
Fingerprinting an OS:
The last scanning method is called Fingerprinting. Fingerprinting is the technique of interpreting the responses of a system in order to figure out what it is. Unusual combinations of data are sent to the system in order to trigger these responses. Systems respond the same with correct data, but they rarely respond the same way for wrong data.