Harvey - the most powerful hurricane to strike Texas in more than 50 years when it came ashore on Friday - dumped more rain on Houston on Monday, worsening the flooding that has paralyzed the country's fourth-largest city.
U.S. crude futures CLc1 dipped by more than 3 percent to $46.21 per barrel over concerns that the refinery shutdowns could reduce demand for American crude.
The uncertainty pushed investors toward safe-haven assets, with gold rising to its highest in more than a week and U.S. Treasuries holding steady ahead of employment data on Friday.
"What it tells you is a slight risk-off trade. But nothing that is really shaking up the markets," said Matt Lloyd, chief investment strategist at Advisors Asset Management.
"We're looking at this as more of a shorter term phenomenon and the energy stocks have held in relatively well," said Matt Miskin, market strategist at John Hancock Investments.
"The market is not seeing that as a significant move but we'll have to see how the dynamics play out over the course of the week."
Nine of the 11 major S&P 500 sectors were lower, with the energy sector's .SPNY 0.91 percent fall leading the decliners, followed by financial index's .SPSY 0.67 percent.
Flood damage from Harvey may equal that from 2005's Hurricane Katrina, which resulted in more than $15 billion in flood insurance losses, according to an insurance research group.
Home Depot (HD.N) gained 1 percent, while Lowe's (LOW.N) rose 0.71 percent as investors expected the two largest U.S. home improvement chains to be among the biggest beneficiaries of post-Harvey recovery.
Kite Pharmaceuticals (KITE.O) surged 28.40 percent after Gilead Sciences agreed to buy the immunotherapy developer in a deal valued at $11.9 billion. Shares of Gilead were up 2.27 percent.
Expedia (EXPE.O) fell 4.11 percent after an internal memo by the online travel services company said its CEO, Dara Khosrowshahi, has been asked to lead Uber [UBER.UL].
Declining issues outnumbered advancers on the NYSE by 1,635 to 1,160. On the Nasdaq, 1,434 issues fell and 1,350 advanced.
Reporting by Sruthi Shankar in Bengaluru; Editing by Sriraj Kalluvila