Not a bad idea to have dual NICs if you are planning on having lots of network traffic. Hard part might be how to share only certain things over a specific NIC. Pretty sure you can do it with VLANs but it takes a bit more work.
My thought is why not just get a dual port Intel NIC that can do teaming. This is what I use in my server. The two ports get teamed and have the same IP address. Then you can set it up to spread the load across the two ports. So you can have one port being maxed out doing a file transfer and the other port would automatically service any other requests. The nice thing about the Intel cards is you can do all of this in the driver and don't even need a special switch. Typically you can find the Intel PRO/1000 PT Dual Port Server adapters for around $30-$40 on ebay. Actually the HP NC360T is a HP branded version of this card and can sometimes be found cheaper. http://www.ebay.com/itm/HP-NC360T-Du...item20d0003a92
The only downside is these cards will only fit in a PCI-E x4, x8, or x16 slot. Other than that they work great.
Hope that helps.