To be compatible across different backends the best option is to store ips and netmasks as longint and manage the transformations in code, otherwise, if compatibility is not a concern or writing that code is a problem, both mysql and postresql offer their own built-in functions to aid with the job.

The problem

The proposed representation is valid for any operation on ips but for the sake of this article I'm only going to consider the simple case that spawned this post. Given a table storing a list of hostnames, their ip and netmask, list all hosts in a given subnet.

The backend agnostic way

Store ip and netmask as unsigned integers doing the transformation in code, either by using functions like inet_aton that most of the programming languages provide, or by applying the following rule: (16777216*a) + (65536*b) + (256*c) + d , ip address being a.b.c.d.
Then, given a network address and a netmask in input (if you get a CIDR notation you'll have to do double transformation), apply the same as above to convert ips to int notation and then you can do:
SELECT * from ips WHERE ip & netmask = $network_addr & $netmask

The mysql way

Mysql provides inet_aton() and inet_ntoa() functions so you can skip all the conversions (except the for CIDR transformation) and store your ips as strings, which makes it much easier to read if by any chance you need to take a look at the DB directly. If you do lots of calculations and never print out ips, for efficiency (although I haven't measured the speed benefit) you can still store ips as int and use inet_ntoa when doing manual selects.
Also, if you choose to use string representation, bear in mind what's gonna happen when you want to sort your list.

The postgresql way

As usual postgresql offers way more than mysql when it comes to features and has a vast range of network type functions (it can even deal with CIDR in SQL).