Minimizing the "Skim" -- the Key to Retiring Early

I came across this 13 minute video by comedian David Cross Why America Sucks at Everything a few days ago. It perfectly summarizes what I'm going to cover in this article. America is organized to screw people who work for a living for the benefit of inherited wealth and those getting their incomes in qualified dividends and capital gains. You want to remove yourself from the group getting screwed, and join the group benefiting from the screwing, as quickly as possible.

The "Skim" is the excess fees, commissions and costs that conventional wisdom, or great sales and marketing, causes many people to pay. If you minimize the "skim", you can retire even earlier. The Big 5 areas of "skim" for most people are 1) taxes, 2) health care and health insurance costs, 3) investment fees, commissions, and trading costs, 4) housing and real estate costs, and 5) college costs.

#1 Taxes-- What gets taxed, and what gets subsidized, is the truest measure of a nation's priorities.

In America, we tend to tax working people and small business owners up the wazoo. Inherited wealth gets a pass (e.g., America's large multigenerational fortures are mostly the result of the lack of taxation, rather than whatever productive activity the first generation workers were up to) and qualified dividends and capital gains are taxed at a much lower rate than wage and salary income.

Case in point: a married couple with a $100,000 annual wage & salary income, taking the $24,000 Standard Deduction, would pay $8,722 in Federal Income Taxes for 2021, plus another $7,650 in FICA. A neighbor couple with a multi-million dollar investment portfolio might decide that working made little sense. Instead they draw $100,000/yr in qualified dividends and capital gains from their portfolio. Their Federal income tax liability? Zero. (See, 2021 IRS Federal Income Tax Table)

FICA punishes working people even more. You secure your full Medicare benefit with just 40 quarters (10 years) of work. You gain no additional Medicare benefit from FICA taxes paid beyond that. For a person with an income triggering the maximum FICA tax, you get very little benefit from working past the age where your Average Indexed Monthly Earnings (AIME) exceeds the 2nd bend point of the calculation. That would be about age 45 for someone paying max FICA starting in their mid to late 20's. If you happened to retire right at the top of the 2nd bend point, you'd be getting 77% of the maximum monthly Social Security benefit while only paying about half the FICA taxes. There's not much of a case for working beyond that. (See: Why high income 40 year old retirees still get crazy big Social Security checks at age 62 )

When I quit my high salary engineering job back in 1994, I was astonished at how little I paid in taxes on the same level of spending. You almost had to volunteer to pay any taxes by selling something. If more working folks understood this, they'd be sharpening their pitchforks. The dumbest thing you can do in America, tax-wise, is to work for a living earning wage & salary income.

#2 -- The Health Care "Skim"

Americans are forced to spend twice as much money on health care than the residents of any other large, industrialized nation, but receive poorer results as measured by life expectancy. Economist Angus Deaton describes the $8,000 extra paid by the average family due to various forms of bipartisan political corruption and outright price gouging as a "poll tax".

There's not much you can do about it if you're working and earning wage & salary income. However, as an early retiree living off an investment portfolio, you don't have to offer yourself up for the fleecing.

The Affordable Care Act (i.e., Obamacare) doesn't have an asset test for eligibility -- it only looks at income. You could have $100 million in Berkshire Hathaway stock and still qualify for a big refundable tax credit under the Obamacare law. It's actually quite easy to minimize taxable income by planning ahead and arranging your portfolio to focus on capital gains and returns of capital while limiting interest & dividend income. See link: Obamacare makes it easier for millionaires to retire early?

When I was doing my long term planning for early retirement 27 years ago at age 38, I expected to be paying about $20,000/year for health insurance by the time I turned age 60. I actually paid less than $20/year in health insurance premiums the last few years before I turned age 65 and qualifed for Medicare in February 2021. See link: Obamacare Repeal? My amazing story of drastically lower premiums.

Like most Americans, I still had a crappy health plan with high deductibles and co-pays that left me vulnerable the the depradations of "out-of-network" billing if I stumbled into the care of an unprincipled doctor or hospital administrator. But at least I was paying a premium more in line with what the for-profit, private health insurance was worth.

The other thing that should anger people is the fact that our political leaders decided to launder the Obamacare tax subsides through the private health insurance industry. There they skim off at least 20% of it in overhead, captive PBM price gouging, and outrageous Executive Compensation. It would be much cheaper to provide the health care through Medicare/Medicaid. And the Gov't bureaucrat who runs the $1.5 Trillion a year Centers for Medicare and Medicaid Services earns an annual salary of $165,000 and flies commercial. Who's giving you the better deal?

#3 The Wall Street "Skim"-- If you're paying the 2% of assets in annual fees, expenses and trading costs that Wall Street's business model demands, you need to save about twice as much money for retirement.

Retirement is a long term, 50 or 60 year project. For most, it will mean 30 years of savings, hopefully followed by 30 years or more of spending in retirement. Compounded investment returns over the long run are going to be a big part of your success. See link: The 2% Rule for Retirement Savings

I've always paid close attention to what I'm losing to fees, commissions and costs. When my portfolio was mostly long-term buy and hold stocks and I rarely made more than 1 or 2 trades per year, that gave me annual investment expenses of 1 or 2 basis points per year (i.e., $100 to $200 per $1 million invested.) Today, I have about one-third of my assets in index funds, so my costs are a bit higher at 4 or 5 basis points overall.

Even after 27 years of annual withdrawals, my retirement portfolio has still grown enough to make my spending in retirement less than a 0.50% of assets annual withdrawal. In effect, I'm comfortably living on the money most people are losing to a financial advisor or mutual fund manager.

As the table below illustrates, even a low-fee advisor charging a 0.30% of assets fee like Vanguard's Personal Advisor Service will claim about 10% of your wealth over 60 years. Those advisors taking higher fees will claim significantly more of your wealth -- like more than half of it.

#4 -- The Housing and Real Estate "Skim"

The standard conventional wisdom that most of us are taught from an early age is that renting "only gives you a stack of receipts, while buying a home builds wealth." And there are wonderful tax deductions homeowners enjoy that are denied to renters. Unfortunately, as investments go, single-family real estate tends to be a poor one. No one ever tells you that.

Nobel Laureate Economist Robert Shiller notes that US single-family housing has lagged the stock market by a wide margin over the past hundred years. The Case-Shiller Housing Index is but a small fraction of the value of the S&P 500. Over the past 100 years, for every dollar you earned in real estate, you would have $17 if invested in the S&P 500. For the past 10 years, every $1 earned in real estate would have returned $2.80 in the S&P 500. Thus, it's wise to do a "rent vs. buy" analysis to inform your housing decisions. Your 20% down payment will likely do a lot better in an index fund than tied-up in underperforming real estate.

Over the past 40 years, I've largely viewed real estate as an expense to be minimized rather than an investment. The rent vs. buy calculation didn't turn positive for me until 2012 when I was living in a Portland OR suburb at the tail end of the 2008 housing crisis and bought a foreclosed home for 70% off its 2008 value. It's since more than tripled in price. That's the kind of unicorn you need to find in the residential real estate market to get anything like an S&P 500 return. And about the only thing that creates those kinds of real estate "gems" is the widespread pain and suffering of a massive housing market collapse.

When I moved to Houston from New York in 1981, I rented a 600 SF unit in a large garden apartment complex less than a mile from the Galleria Shopping Center for $400/month. The owner was in the process of turning half the property into condos. A 600 SF condo was selling for about $45,000 at the time. Mortgage interest rates were around 14% in 1981, but with itemized deductions and a high enough salary, after tax, the monthly cost of owning was about the same as renting.

Today (2021) that 600 SF Houston Galleria area condo sells for about $100,000 and the 600 SF apartment rents for $850/month. If you put the 20%, $9,000 down payment on the condo in an S&P500 index fund over the past 40 years, you'd have $693,000 today and the 1.6% dividend yield (about $11,000/yr) would more than cover the $850/month apartment rent.

I lived off and on in Houston for 25 years before I fled Texas for Washington State in 2006. It was a marvel to observe the low housing costs in the city. As REITs continued to build new apartment complexes as far as the eye could see, my monthly rent barely budged. When I left town, I was paying less than $600/month for a 900 SF unit in an apartment complex with several pools and tennis courts. I guess real estate developers are going to develop, and builders are going to build, as long as investors keep throwing money at them.

The biggest obstacle to real estate returns are the transaction costs. I can sell a $1 million block of stock for a 0% commission and perhaps a 2 cent bid/ask spread on a heavily traded stock like PFE (i.e, 20,000 shares @50/share x 2 cents/share = $400.) Where I live in WA State, it would likely cost at least 10% or $100,000 to sell a $1 million home if you're paying a 6% commission, the 1.7% state franchise tax on real estate sales, plus the usual closing costs like title insurance, escrow fees, etc.

#5 - The College "Skim"

When I got my engineering degree way back in the mid 1970's, I attended what is now a very expensive private college (WPI: 2021 Average cost before aid, $72,296 per year). But it was a very different story back then. WPI cost about $5,000/yr. I graduated in three years due to the school's innovative curriculum at the time that focused on project work and passing a comprehesive "Competency Exam" in your major. The starting salary in my first job was a bit less than the cost of 3 years of tuition and room & board. Being able to earn that big engineering salary a year early almost made the education "cost-free".

Today's students see a new reality. Private college costs have ballooned far beyond what's reasonable based on the salaries and likely lifetime earnings of new graduates. And in many programs, the fancy private college degree doesn't deliver enough in additional salary to warrant the $15,000 to $20,000/yr added cost versus in-state tuition at a public university. No way I'd take out an additional $80,000 in student loans unless I was sure I'd be earning enough extra income to pay it off. And if my family was wealthy enough to just write a check, I'd be evaluating if that additional $80,000 might be better allocated to an index fund. The 40-year return on that might well secure our new graduate's retirement.

Fortunately, there are tools available today that allow students to better measure the economic value of the various schools and their degree programs. Don't get hoodwinked into "college at any cost". The US Education Dept has a useful database that allows you to compare a college's cost with the starting salaries for new graduates.

As much as I value my WPI education (especially that 3-day financial planning course I took as an 18-year-old Freshman that clued me in on the fees and costs financial advisers and insurance salesmen were taking from their clients), I'd have a hard time recommending the school to a young person at current pricing.

What to conclude from these results?

While there's no harm in using coupons at the grocery store, or forgoing Starbucks for instant coffee at home, the big-impact savings are in the big ticket items.

Doing the "rent vs. buy" analysis over the years to inform my real estate decisions saved me at least a million dollars. And that savings compounded into millions more since it was invested in the stock market over two or three decades rather than a poorly performing "investment" in single-family real estate. Obamacare didn't happen until I was almost age 60. My early retirement was more than secure by then, so the $100,000 savings was just icing on the cake. Still, it's better to have that windfall in my pocket than skimmed off to an insurer's Executive Compensation.

But by far the biggest advantage I had was early knowledge of the affect of investment commissions, fees and costs. Most people seem to make a few mistakes in the early stages of their financial life and lose a decade or two of compounded investment returns before they realize they've been screwed. I was able to avoid most of that and retire early on the strength of that 3-day financial planning course I took in college. It was by far the most valuable piece of information I gleaned from my engineering education.

Resources for more information

Should You Buy a Home in the US? Robert Shiller, October 25, 2021.

Retirees' Bad Arithmetic Reduces Social Security Trust Fund Withdrawals by $3.4 Trillion -- Making the Trust Fund Last Longer REHP, July 1, 2019.

The Growing Gap in Life Expectancy by Income: Implications for Federal Programs and Policy Responses (2015) -- The National Academies of Sciences, Engineering, and Medicine

Cadre Rent vs. Buy study -- Third Quarter, 2023

Bengen, William P, �Determining Withdrawal Rates Using Historical Data�, Journal of Financial Planning, October 1994, pp 171-180, Volume 7, Number 4.

Investment Company Factbook 2019, ICI.org