And now for a bit of a technical aside — my regular readers will probably want to skip this post as it has little to do with regular gamedev news from me.
However, I find myself in the situation of needing to document a technical solution, and having nowhere else to post it, will sneak it into this blog. Mostly so next time I stumble into this, I can quickly look up the solution instead of re-solving it again and again. (Yes, I am absolutely the kind of dev who searches for a technical solution, only to find and follow my very own post from a few years prior :P)
((And also because I don’t have enough reputation points to post this answer on Stack Overflow — ironically, too many people there have this question, which has increased the rep required to post an answer… meanwhile all the answers are wrong/outdated, leading more people to have this question…. Ah, classic Stack Overflow, lol))
Background – Subversion
Subversion is a version control system used by some developers to store game assets during development. Most devs have moved on to using git or perforce, but svn is still used by a few indies (like myself) thanks to its ease of use and binary file handling. That said, even I have been using git more frequently lately and have been eying the ongoing development of LFS. Subversion is not without its quirks, and I have had my fair share of subversion corrupting itself randomly. To be fair, I’ve also seen the very nice, but very expensive Perforce software corrupt itself. Meanwhile, svn is good and free so… I’m still using svn (with lots of backups and automated checks! Actually, I should probably document that script sometime, too…)
Background – Plaintext & Machine Credentials
Question: When is it okay to store plaintext / unchallenged credentials?
For storing a human’s credentials (ie, user databases, etc), basically never. (In fact, I use strong one-way hashing for this to safeguard passwords securely)
But machine credentials work differently than human credentials. Basically, servers have automated processes. A server needs to be able to use its credentials, even when a human isn’t there to type them in. Example: SSL certs on web servers — the web server needs to be able to read the cert’s private key whenever it starts up. If it can’t, the website won’t start and it’s offline.
Or in today’s case, a build server that needs to automatically login to a version control software in order to make automated builds for a game. In that situation, you absolutely wouldn’t want to have to walk over to the build server and retype a password over and over again. It’s designed to be fire & forget. Check in code, wait, get builds automatically.
Unfortunately, in recent versions, Subversion makes this quite difficult as the developers have conflated human credentials with machine credentials.
Problem
Due to some crazed moon logic, the subversion developers have decided to make support for plaintext passwords a compile-time option. While this might make sense for desktop installs that have keyrings, it very much does not make sense for automated build servers and some other CLI environments. In the aforementioned link, I already commented at length on how astoundingly foolish this decision is, but it’s two years since that post and as svn continues to dwindle ever lower in usage (for this and other reasons), the developers don’t seem to want to budge. So, assuming you don’t want to recompile + maintain svn from source (and you probably don’t!), now we are firmly in workaround territory.
Workaround #1 – ZSH Script
The ‘compromise’ subversion developers are now instituting is that even if subversion was not compiled with plaintext support, you can still crowbar it in by using an external tool. Basically, the external tool will write plaintext passwords, and all versions of subversion (even without compiled support for it) will be able to read plaintext passwords. (I have no idea how they decided that this is somehow better than the old method of just using a run-time override flag ¯\_(ツ)_/¯ )
This script by Daniel Shahaf was posted in the thread as a workaround:
#!/usr/bin/env -S zsh -f
# Prompt for a realm and a password, then cache that password for that realm, in plaintext.
PS3="Enter the number of the selected option: "
creds=( "${(ps.\n\n.)"$(LC_ALL=C svn auth)"}" )
creds=( ${(M)creds:#-*} )
select m in $creds
do
realm=${(M)${(f)m}:#Authentication realm: *}
realm=${realm#*: }
IFS= read -s -r pw"?Password: "
md5=${"$(printf %s "$realm" | openssl md5)"##*= }
print -rC1 \
\$ i \
"K 8" passtype "V 6" simple \
"K 8" password "V ${#pw}" "$pw" \
"." \
"w" \
"q" \
| ed -s ~/.subversion/auth/svn.simple/$md5
echo edited $_
break
done
And this process seems effective for usage:
Do an initial test download from the repo (ie svn co https://user@location/repo dest)
Note that every time you try to update, you get password prompted
Dump the above script into a storePlaintextSvn.zsh file
apt-get install zsh if you don’t already have zsh
Then run zsh storePlaintextSvn.zsh
Select the repo (you’re looking for a ‘svn.simple’ entry)
Enter password
Now re-try the svn update, and it should work without badgering you for a password
Now you can do whatever svn build automation you were working on without manual human intervention.
I tested this on Ubuntu 20.04 and it seems to work so far!
Workaround #2 – Python Script
The FAQ on this very issue also now mentions a Python script forthcoming. It looks like it will be located here, though as of this writing it’s not actually linked yet.
This is going to sound like the dumbest reason to delay a game launch, but it’s 2022 and algorithms now control our lives in the stupidest of ways, so yeah… Steam launch is delayed due to gmail.
It starts with wishlists…
My expression when dealing with more big tech shenanigans
Kobolds is basically done (a few people have already bought and played it direct — sincere thanks if that’s you :)). There’s still the Steam integration and packaging up the bonus content, but that should be pretty fast.
The much, much harder challenge for a Steam release is to build up wishlists. Industry pundits frequently say that in order to get meaningful visibility on Steam, you need between 10,000 ~ 50,000 wishlists. That is basically impossible for smaller developers, but there are plenty of anecdotes floating around that even if you don’t hit that threshold, the more wishlists you have before your steam launch, the more likely it is that Steam will show your game to other players.
Alternatively — don’t get enough wishlists, and your game is essentially buried. And I know buried — my last game was so buried on the Steam store, that for a while it didn’t even show up in top search results for an exact name match.
It’s a broken dynamic, since the whole point of platforms is that they’re supposed to make it easier for developers and players to find each other. The reason developers accept the outrageous fees for these platforms is that, in theory, you gain more sales than you otherwise would get. But, for small developers, that’s not happening. If you’re small, none of the platforms do discoverability anymore. It is a truly dysfunctional world in which small developers are expected to send traffic to Steam, which already has an absurdly large amount of traffic. But, that is Steam in 2022 — send traffic, or have a do-nothing Steam launch.
Wrestling with Gmail
So, where does gmail come into play in all this? Well, for building wishlists, there are many strategies. For some games that are highly replayable, streamers can be a big boost. Kobolds isn’t really a great game for streaming as it’s a linear play-once adventure game, but there’s still other methods.
Like most developers, I have a newsletter where I send out news about my games launching. It is comprised almost exclusively of people who have played my games previously (and also a handful of interested press persons and such). Last week, I sent out a mail about Kobolds being now available on the store, as well as being wishlistable — for many fans, it was the first time they had heard from me in a year since my last game came out.
Unfortunately, almost no-one with a gmail address received that e-mail. Leading me to my main point in this post/rant: it is astonishingly difficult to send e-mails at scale, even if you’re not a spammer.
We’re a good sender
Let me be clear – we’re not a spam sender. People opt-in to receive our mails. This isn’t an unsolicited message. Over the years, I’ve had many variants of questions from fans who (fairly) complain that they signed up but didn’t receive the newsletter. But gmail is treating us like a spammer regardless.
Our IP isn’t on the main blacklists commonly used by ISPs (we’ve had exclusive use of it for years now), and Google’s own tools show that we haven’t received a single human complaint for our recent message (for the few people that got it):
No spam here
And yet, our reputation is considered ‘bad’ — likely due to the fact that we only occasionally send batches of mail out instead of at a constant pace — a subtle AI rule that both favors other platforms instead of small business directly as well as makes it much harder to send out large message batches in a timely fashion (such as before a game launch).
No spam reports… and yet, bad reputation??? >.<
When you have only occasional messages to send, gmail tends to confuse you for a spammer. There’s mitigation strategies, like ‘warming up’ your IP to ramp up sending — something that I did, but either it was too ‘cold’ that even that ramp was too fast, or their AI is just broken in a different way (google, of course, won’t just tell you what’s wrong). There’s all sorts of other sub-strategies that e-mail experts basically say you have to use to get around this to contact people who requested to be contacted, and well, it’s a massive headache and time sink.
There are so many obstacles that supposedly are designed to prevent spam, but seem to be designed more to discourage small business from using email to keep in touch with their contacts. For instance, take the case of receiving a message that ended up wrongly in spam. You might think clicking g-mail’s ‘not spam’ button would add that sender to a list of safe senders. NOPE. Messages from that sender will still end up in your spam folder unless you follow a convoluted process that almost no one is actually going to do. Google knows this, it’s a dark pattern. They don’t want businesses to send newsletters to their customers, they want those businesses to buy advertising.
“No e-mail for you!”
Getting e-mails delivered has become really an obtuse process. There’s plenty of technical jargon senders have to understand and do, and even then you get into situations where e-mail gets blocked. Sure, you can hire a service to help a lot of this for you, but it’s not really affordable at scale for indie devs. And then you’re right back to where you started, relying on platforms.
The worst problem offenders are the two largest networks – gmail and hotmail. They both have a tendency to block or filter mails based on faulty AI. In my experience, sometimes hotmail is worse, sometimes gmail is worse, but since gmail addresses are 60% of my mailing list, I’m kinda forced to dance to their tune if I actually want to send messages.
My IP Blacklist search results are good overall.
Here is really the crux of the problem — the major e-mail platforms all also run ad networks. It’s a massive conflict of interest because it incentivizes them to take steps to make e-mail harder. If they can either prevent or raise the cost of communicating with your existing customers, it incentivizes businesses to just buy advertising instead. They make money on ads, not on email. In what I’m sure is a coincidence, I’ve gotten unsolicited spam from both Microsoft and Google (that I did NOT sign up for) trying to get me to buy ads.
Gmail makes e-mail worse for everyone
The major email providers (especially Gmail and Hotmail) set policies that actually lead to making e-mail worse as a whole. You get into situations where businesses are pressured to send pointless e-mails just to keep lists ‘fresh.’ If I only have something important to say once a year, I should be able to just send one email a year. But since the algorithms favor ongoing constant-rate sending, it applies needless pressure to come up with things to message people about on a more frequent basis or you end up in the exact situation I’m in right now of being blocked. I deal with it anyway because I’m commited to only sending important messages, but we’ve all seen other people’s newsletters full of useless fluff, and that’s the reason.
‘IP Warming’ in action
Another dumb policy — small businesses are de-facto required to snoop on who is opening or not opening messages, because gmail uses message opens as a major signal for if a sender is ‘spammy’ or not. Low opens = spam, high opens = not spam. Which means senders are supposed to prune out people who aren’t opening messages, which means you’re expected to creepily spy on who is opening messages. This is also the reason you get those dumb ‘do you still want to receive messages from us?’ spam that so many lists do — because gmail literally forces you to only send to active readers or they give you a ‘bad’ reputation which leads your messages to go into the spam folder or be blocked entirely. I’ve had to prune hundreds of thousands of people off the newsletter over the years just to play these many weird games that gmail forces us to play, and it sucks. Forcing people to nerf their lists, spy on people and generally making e-mail announcements useless or hard is an absolutely intentional big-tech strategy on their part. It’s the same thing as Facebook when they started limiting page reach for businesses unless you paid up. It’s classic ‘create the sickness, sell the cure’ business.
The e-mail gmail doesn’t want you to see
Google doesn’t even follow their own rules – at one point they were automatically signing up existing accounts to receive spammy notification mails from youtube, which is against their own sending policies. My own policies, which require user consent before subscribing them and also provide a one-click unsubscribe link and relevant unsubscribe headers, actually beat those of Google. But as usual with big tech, it’s one ruleset for themselves, and another for everyone else.
It’s especially ironic since when I receive personal mail, gmail scammers are one of my biggest sources of incoming spam, and there’s no way to contact an abuse e-mail address to report them to Google. Supposedly there’s a web form somewhere, if you can find it — but a provider as large as google should absolutely have an abuse contact so people can report actual spammers on their network right from their mail client.
It doesn’t have to be this way
Yes, e-mail as a whole does have a spam problem, but it seems like there are obvious, easier solutions to certain elements of this rather than coming up with infinite hoops for legitimate senders to jump(/stumble) through.
E-mail providers should be able to remember what their users do with mail. If you mark a sender address as either ‘spam’ or ‘not spam’, Gmail should actually remember that for the lifetime of your email account (even marking spam has a hidden limit). [Update – similarly, hotmail seems to have a forgetfulness problem. Even though one of our fans marked us as a safe sender, hotmail magically forgot this over the intervening years and the newsletter ended up in their spam folder]
That memory alone would make sending so much easier for legitimate senders. Instead you’re fighting the shifting winds of algorithms, and one of the reasons I want to use e-mail instead of social networks is to avoid having to constantly capitulate to algorithms. Nowhere is safe from big tech trying to get in between content creators and their fans, and that sucks.
To be fair, I realize this is just one element in the much larger puzzle of spam/ham detection. Solving the ‘new message from sender we haven’t seen before’ or ‘valid sender but they got hacked’ or any number of other cases quickly illustrates how deceptively complex the problem of e-mail is. But if Google could quickly fix deliverability problems with human intervention instead of making senders wait weeks a lot of this wouldn’t be as big of a deal as it is. It’s not even obvious where their postmaster contact form is from the link on their error message — I had to find it via DuckDuckGo.
Closing remarks
E-mail was one of the last bastions of the free web, but the major platforms have made it so hard to get legitimate mail delivered that it’s essentially just as platformized as the rest of the web now.
So, anyway, I’m going to take some extra time to try to fight GMail on this and try to get the launch announcement through their filters. Most people seem to just give up when platforms put obstacles in the way — and maybe that’s the smarter strategy to be honest — but it leads to infinite platform hopping like we’ve seen (on that note, how long until Discord monetizes the @everyone ping? I’d put money it’ll happen sooner or later. Every other platform finds ways to put obstacles between creators and fans, and then monetize the ‘solution’ to their own obstacles).
Will these efforts result in an appreciable number of wishlists for Kobolds? Probably not. The way Steam works now, I don’t see any viable way for a small dev to hit the kind of thresholds needed for serious organic discovery. But every wishlist does help a bit, and at least it gives the game the best shot it can have in this big tech platformized dystopia we’re all forced to live in now.
