FORTUNE — The very people who should care most about J.P. Morgan’s shareholder votes on Tuesday may be people who don’t own a single share of the bank’s stock.

They are the people who saw their savings wiped out by the derivatives and securitizations boom that fueled the last financial crisis. They are the people who lost their jobs and are still looking. They are people buried in paperwork on bank debts they don’t owe. And they’re the new grads who haven’t found a job worthy of their education because the economy is still regaining its feet from the biggest recession since the great depression.

These are the people who need J.P. Morgan (JPM, -0.12%), all banks, and all companies to be governed well — because they can’t afford for them not to be. Bereft of trust funds from their parents, they rely on a strong economy.

Both this year and last year, a shareholder proposal to split the CEO-chair positions at the bank appears to have won higher percentages than similar measures at other S&P 500 companies, according to data from proxy advisory firm Institutional Shareholder Services (ISS). But, based on preliminary information, the proposal did not receive majority support this year.

It does appear that the fight wasn’t fair this year. Nearing the election, shareholder proponents were denied information on the vote’s status for a period of time. The Council of Institutional Investors (CII), which represents investors with assets in excess of $3 trillion, sent a letter to SEC chief Mary Jo White on May 17 asking her to review the matter. J.P. Morgan cut off discussion on the topic at the annual shareholder meeting on Tuesday, leaving shareholders with no satisfactory response, according to Michael Pryce-Jones, senior governance policy analyst at Change to Win investment group. Due to the lack of even-handedness, at a minimum, an audit of the votes in this case may be in order, to supplement the review CII has requested.

The lack of majority support for the split delays what even board members with combined CEO-chairs now see as an inevitable trend away from combined roles. A board member of several technology companies told me on Tuesday that there’s no reason not to split at any company. “It’s just not possible for someone to effectively wear both hats,” he says.

While short-term shareholders may cheer J.P. Morgan CEO Jamie Dimon’s retention of the chair spot, his position in dual roles will drag out the process of developing a strong board and harm the board’s near-term ability to rein in risk. Pryce-Jones says the meeting demonstrated both Dimon and J.P. Morgan presiding director Lee Raymond’s lack of understanding of the risks related to the bank’s PetroChina investment and that they were unable to answer questions on the topic, although shareholders have raised the issue in 2011, 2012, and 2013. According to a CNNMoney report, CLSA bank analyst Mike Mayo said about the votes, “There’s some significant regulatory tail risk for J.P. Morgan, and that’s not going away. I’m not sure the regulators view the decision today as favorably as the stock market currently does.”

Three J.P. Morgan board members received strong rebukes from shareholders on Tuesday. The three directors joined just six other directors in the S&P 500 this year that have received over 40% of votes against their reelection, Pat McGurn, special counsel at ISS, told me. CtW Investment Group is calling for all three bank board members to step down due to a “resounding vote of no confidence.” J.P. Morgan didn’t respond to a request for comment prior to publication of this article.

Comparing what happened at J.P. Morgan this week with the shareholder votes at other S&P 500 companies, the bank board received significant votes of no confidence on both its structure and its members. But looking at the raw vote count, it illuminates the concern that too many shareholders are short-timers or are conflicted.

Large institutional investors without courage often find themselves unable to vote against their peers. The potential loss of business that may result from opposing votes can cow them. And it’s sometimes hard to vote against someone like yourself, an issue boards face when a sitting CEO is on the board’s compensation committee. In advance of the financial crisis, several large active shareholders told me that they would not engage on governance matters with other financial institutions because “they are too much like us.”

So where is the accountability — and where does this leave our country with too-big-to-fail banks that the Justice Department is afraid to prosecute? Clearly our system needs repair.

We do need to find ways to check cronyism and institute better governance. If we don’t, we will find ourselves continuing our race to the bottom — and citizens without trust funds will continue to be the fall guys.

Eleanor Bloxham is CEO of The Value Alliance and Corporate Governance Alliance (http://thevaluealliance.com), a board advisory firm.