How to do an initial push of a very large git repo

I've been trying to mirror the linux code repository from one git server (e.g. Gerrit) to another (e.g. GitHub Enterprise) since yesterday. The theory was nothing more than cloning from A and pushing to B. What makes this case worth a blog post is that linux is a project with decades of history (I just learned that this is not even a complete one) and millions of commits. GHE, on the other hand, limits the upload size. So when I did the push step (after cloning the original repo, renaming the original remote to upstream, adding the new repo as origin), I ran into the following error:

fatal: The remote end hung up unexpectedly
error: pack-objects died of signal 13
error: failed to push some refs to 'git@github.xxxx/XXX.git'

I then Google'd a bit and found a possible solution. The nice one-liner that javabrett mentioned splits the whole commit history into numerous pieces, each OK to be pushed, then pushes them one by one.

The paging parameter, i.e. number of commits each piece contains, needs to be carefully crafted. If you set it too high, you end up with fewer pushes, but one of them might exceed the size limit. Setting it too low, you'll then have to bear with more pushes, thus more waiting time. It's a trial-and-error kind of experience before you could finally settle. The silver lining, however, was that you could start where you failed last time, without deleting the repository and doing it all over again - git is smart enough to reject existing commits from being pushed. The one I finally settled with is 5000 (commits per push):

git log --reverse --oneline | sed -n '0~5000p' | awk '{print "git push origin "$1":refs/heads/master"}' | while read i; do eval $i; sleep 2; done

Not every push in the whole process was a success. Sometimes I ran into:

To git@github.ibm.com:workload/linux.git
 ! [rejected]        fdc657c -> master (non-fast-forward)
error: failed to push some refs to 'git@github.ibm.com:workload/linux.git'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

I suspect this was because I did everything in a loop and the next push came in before the server could fully settle the previous one (even with 2 seconds' sleep). I didn't actually look into the errors because commits in a failed push would also be carried in subsequent pushes, as long as one of them succeeded.

After the one-liner loop was done, the remainder commits needed also to be pushed:

git push origin master

Also if you have other branches, push them directly (maybe a --dry-run before the real deal) as they share most history as the master branch and the push would not be too large.

git checkout branchA
git push origin branchA

To double check if everything is in place:

git push --all origin

You should see Everything up-to-date.

Also don't forget tags:

git push --tags origin

Last but not least, you can do it from a whole different perspective, because only recent commits matter to you, say, one year. One from 2008 doesn't. So a shallow clone with reasonable depth should be just fine.

流水账之2016北京车展（上）

回到家，洗完澡，烧好了水，衣服扔进洗衣机，坐下来，开始写点东西。

我没有可以计步的装置，不过一天下来几万步总是有的吧，仅凭着一个面包和600ml的热水。真的很累，腿脚都生疼。头脑却异乎寻常地兴奋，完全不应该是昨晚熬夜今早6点多起床一直到现在该有的样子。这个状态像极了之前参加斯巴鲁试驾会玩儿high了的那一天。

缘起

你丫是不是有病，一个流水账还值得写缘起。好像是有些故弄玄虚了，不过我还是想写写。之前总是有事没事和老婆提起，说想去看车展，看她能不能想办法在媒体日把我混进去。某次她笑我，说人家都是有媒体证的，你是个啥媒体呀。我当时心一横，脱口而出说我是自媒体。她又问说那你都发表过啥。我说博客啊，只不过被墙了没人看到过。

今天在看车展的时候，老婆发来微信说要不你真的写点啥吧。好吧，人不能太懒，那就写写。

吐槽

如果谁要是万中有一在赴车展之前看到了这篇文字，如果他还没有在网上订票，那么你听我的，去现场买吧，人（买票的）不多，而且组织有序，至少今天是这样。

去看车展的想法由来已久了，不过一直也没有想好到底哪天去；是周末去，还是工作日请假去。大概三四天前，看到YYP的公众号说要27号在车展现场做一次真人版Ask YYP，心动了，那就这天吧。

然而当我准备在车展官方指定的由易车网操刀的淘宝店里买票的时候，被告知说能邮寄的票已经下架了，只能网上买然后到车展现场取。我彼时还追问过现场取票会不会人很多，对方说取票而已，很快。我又问说到时候你们会不会有明显的标识指示在哪里取票，答曰会。于是我买了。

昨晚在下班的地铁上，收到易车打来的电话，说让今天坐地铁去了之后从D口出来，取票的位置在国展的7号门附近。当时我心想，大厂风范啊，做事周到。然而今天早上当我兴冲冲从D口走到7号门的时候，我看到了爱卡汽车网的摊儿，我好像还看到了汽车之家的摊儿，唯独我没有看到易车的影子。顺着人流从7号门进入，我看到了三五成群一堆一堆的人，我看到了整齐列队的保安，我看到了大牌子指示的售票处，我特么还是没看到易车的影子。我擦，我不会是着了道儿了吧。打电话，买票的时候淘宝小二给过我两个紧急联系电话。挨个打，反复大，没有一个没有一次接通。万般无奈之下我去售票处问，好心的志愿者告诉我说在其中一个售票亭子边上，白色遮阳伞的下面。我手搭凉篷举目远望，果然有个白色的遮阳伞正在搭建。这个时候，9:40，距离开展只有20分钟。

好歹可算是找到组织了。我走到近前发现已经围了三四层人，真正的工作人员，三四个吧；而卖出去的票，名单有好几页A4纸。几位小哥显然也不知道该怎么发票了，让大家自己从名单里找，可是名单只有一两份远远不够分的，等着取票的人却越聚越多。人堆里这时不知道谁喊了一句，顺着名单挨个念吧。小哥们纷纷眼前一亮，对啊。不过好像缺个麦克风哎。于是里面念一个名字，这个名字还要被广大群众一层一层传出来。中间如果哪一层的群众顾不上传了，我猜这张票就永远找不到主人了。此刻的我站在外层，心中真的是有一万只草泥马在奔腾汹涌。看看手表指针已经指向了9:55，看看眼前只增不减的人群，罢了。我很快找到了一个完全没有人排队的售票亭，没错是完全没有人排队，长长一排售票的姐姐都无事可做。买票，安检，进场。这三个动作一共花了不到五分钟，可算是进来了。

至于易车那张没有用掉甚至都没有拿到的票，我已经在淘宝上发起了退款。能退则好，退不了就只好自认买个教训了。