From nompelis at nobelware.com Tue Oct 1 16:33:07 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Tue, 1 Oct 2019 16:33:07 +0000 Subject: Quine - from a Twitter favourite In-Reply-To: <8fa980a5-57a0-4d58-8c35-bf01c0f41764@www.fastmail.com> References: <20190925134937.GA5195@nobelware.com> <8fa980a5-57a0-4d58-8c35-bf01c0f41764@www.fastmail.com> Message-ID: <20191001163307.GB23407@nobelware.com> > > An incredible quine achievement: > https://github.com/mame/quine-relay > Insanity! From nompelis at nobelware.com Tue Oct 1 16:34:51 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Tue, 1 Oct 2019 16:34:51 +0000 Subject: The most evil macro In-Reply-To: <20190928024541.GC4396@begriffs.com> References: <20190928024541.GC4396@begriffs.com> Message-ID: <20191001163451.GC23407@nobelware.com> > > It replaces the builtin "if" function with one that rolls the dice and > performs the wrong branch with probability one in a million. > This is one of the most clever --and evil-- things I have seen/heard done in programming. > We activated the macro, then opened a tetris game inside Emacs (because > of course you can play tetris inside Emacs). Everything was going well, > until the falling block sank into the floor and disappeared from the > screen! And this was a really nerdy way to use it! From nompelis at nobelware.com Tue Oct 1 16:38:03 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Tue, 1 Oct 2019 16:38:03 +0000 Subject: Link Library In-Reply-To: <20190930155832.eut2m5yldiymeucn@begriffs.com> References: <3EAD6468-4877-4E8B-B1EB-9078EAC400AB@gmail.com> <20190930155832.eut2m5yldiymeucn@begriffs.com> Message-ID: <20191001163803.GD23407@nobelware.com> We write a crawler that scavanges links from this list's archives and surprises us one day a week. From nompelis at nobelware.com Tue Oct 1 16:41:32 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Tue, 1 Oct 2019 16:41:32 +0000 Subject: Link Library In-Reply-To: <3EAD6468-4877-4E8B-B1EB-9078EAC400AB@gmail.com> References: <3EAD6468-4877-4E8B-B1EB-9078EAC400AB@gmail.com> Message-ID: <20191001164132.GE23407@nobelware.com> Good fine Dave. There are a lot of gems in this. I just realized that in 1999 DMR was still at Bell Labs. (R.I.P. brothah.) From nompelis at nobelware.com Wed Oct 2 13:43:27 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Wed, 2 Oct 2019 13:43:27 +0000 Subject: Humble Bundle: Unix and Linux In-Reply-To: <20190930205936.GA5466@19a6.tech> References: <20190930205936.GA5466@19a6.tech> Message-ID: <20191002134327.GA11072@nobelware.com> I have had Linux System Programming on paperback for many years. I very very very VERY strongly recommend this book if one is to learn how Linux really works. IT is C-oriented, of course, and an excuse for people like harv lurking here to do some serious damage. From nompelis at nobelware.com Thu Oct 3 02:32:44 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Thu, 3 Oct 2019 02:32:44 +0000 Subject: 2600 going PDF Message-ID: <20191003023244.GA11809@nobelware.com> I hope they keep printing, but we can now get it on PDF with no DRM. https://www.2600.com/content/message-our-readers Long live 2600 and Phrack. From dave.bucklin at gmail.com Thu Oct 3 03:17:14 2019 From: dave.bucklin at gmail.com (Dave Bucklin) Date: Wed, 02 Oct 2019 22:17:14 -0500 Subject: 2600 going PDF In-Reply-To: <20191003023244.GA11809@nobelware.com> References: <20191003023244.GA11809@nobelware.com> Message-ID: On October 2, 2019 9:32:44 PM CDT, Ioannis Nompelis wrote: >I hope they keep printing, but we can now get it on PDF with no DRM. > >https://www.2600.com/content/message-our-readers > >Long live 2600 and Phrack. I did some contracting for a local fintech company a few years ago. I got some legit technical documentation on one of their platforms from an issue of Phrack. In applied research, leave no stone unturned. From drewbenson at netjack.com Thu Oct 3 03:40:39 2019 From: drewbenson at netjack.com (Andrew Benson) Date: Wed, 2 Oct 2019 22:40:39 -0500 Subject: 2600 going PDF In-Reply-To: References: <20191003023244.GA11809@nobelware.com> Message-ID: That?s hilarious. :) Whatever works, I guess! > On Oct 2, 2019, at 10:17 PM, Dave Bucklin wrote: > > > > On October 2, 2019 9:32:44 PM CDT, Ioannis Nompelis wrote: >> I hope they keep printing, but we can now get it on PDF with no DRM. >> >> https://www.2600.com/content/message-our-readers >> >> Long live 2600 and Phrack. > > I did some contracting for a local fintech company a few years ago. I got some legit technical documentation on one of their platforms from an issue of Phrack. In applied research, leave no stone unturned. From nompelis at nobelware.com Thu Oct 3 19:53:24 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Thu, 3 Oct 2019 19:53:24 +0000 Subject: 2600 going PDF In-Reply-To: References: <20191003023244.GA11809@nobelware.com> Message-ID: <20191003195324.GA23974@nobelware.com> Oh, for sure whatever works. Astounding Dave that you actually thought of looking there for info. Or maybe you did a search and it came up. Speaking of FinTech, I watched this documentary below and thought that we if we ever need a new project... https://www.youtube.com/watch?v=kFQJNeQDDHA (Sorry for the tangent) From joe at begriffs.com Wed Oct 9 05:09:46 2019 From: joe at begriffs.com (Joe Nelson) Date: Wed, 9 Oct 2019 00:09:46 -0500 Subject: Weds meeting Message-ID: <20191009050946.3jq3jtr7ocylwau7@begriffs.com> Anyone want to hang out at the hack factory? I'm planning to work on submitting another revision of a postgresql patch to their dev list. If anyone is curious how development happens on the database, you could pair with me on it. From nompelis at nobelware.com Wed Oct 9 15:07:42 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Wed, 9 Oct 2019 15:07:42 +0000 Subject: Weds meeting In-Reply-To: <20191009050946.3jq3jtr7ocylwau7@begriffs.com> References: <20191009050946.3jq3jtr7ocylwau7@begriffs.com> Message-ID: <20191009150742.GA4254@nobelware.com> I will stop by on my way home tonight, at 6:00 or so and stay until about 7pm. Must say hit to my hackgroup. From joe at begriffs.com Wed Oct 9 15:52:26 2019 From: joe at begriffs.com (Joe Nelson) Date: Wed, 9 Oct 2019 10:52:26 -0500 Subject: Weds meeting In-Reply-To: <20191009150742.GA4254@nobelware.com> References: <20191009050946.3jq3jtr7ocylwau7@begriffs.com> <20191009150742.GA4254@nobelware.com> Message-ID: <20191009155226.u7pki7g36qawiupy@begriffs.com> Ioannis Nompelis wrote: > I will stop by on my way home tonight, at 6:00 or so and stay until about > 7pm. Must say hit to my hackgroup. I'll be there a little later (around 7 is my usual time). Although if nobody else says they're planning to go then I'll probably skip it. From dave.bucklin at gmail.com Wed Oct 9 16:58:53 2019 From: dave.bucklin at gmail.com (Dave Bucklin) Date: Wed, 09 Oct 2019 11:58:53 -0500 Subject: Weds meeting In-Reply-To: <20191009155226.u7pki7g36qawiupy@begriffs.com> References: <20191009050946.3jq3jtr7ocylwau7@begriffs.com> <20191009150742.GA4254@nobelware.com> <20191009155226.u7pki7g36qawiupy@begriffs.com> Message-ID: On October 9, 2019 10:52:26 AM CDT, Joe Nelson wrote: >Ioannis Nompelis wrote: >> I will stop by on my way home tonight, at 6:00 or so and stay until >about >> 7pm. Must say hit to my hackgroup. > >I'll be there a little later (around 7 is my usual time). Although if >nobody else says they're planning to go then I'll probably skip it. The Hack Factory open house starts at 7 and I've been asked to wait outside until that time. So you might consider meeting elsewhere before 7. From nompelis at nobelware.com Wed Oct 9 18:27:14 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Wed, 9 Oct 2019 18:27:14 +0000 Subject: Weds meeting In-Reply-To: References: <20191009050946.3jq3jtr7ocylwau7@begriffs.com> <20191009150742.GA4254@nobelware.com> <20191009155226.u7pki7g36qawiupy@begriffs.com> Message-ID: <20191009182714.GA12618@nobelware.com> > > The Hack Factory open house starts at 7 and I've been asked to wait outside until that time. So you might consider meeting elsewhere before 7. Oh! Thanks Dave. OK. IN that case I will skip it. Sorry guys. From zach at nor.mn Thu Oct 10 00:05:38 2019 From: zach at nor.mn (Zach Norman) Date: Wed, 09 Oct 2019 19:05:38 -0500 Subject: Weds meeting In-Reply-To: <20191009182714.GA12618@nobelware.com> Message-ID: <9hrn96m9ldbrp3h6vte7sg94.1570665938215@nor.mn> Kermit and I showed up. Looking forward to anyone else who wants to talk shop, hack the planet, etc Zach normnco automata ? Original Message ? From: nompelis at nobelware.com Sent: October 9, 2019 13:27 To: dave.bucklin at gmail.com Reply-to: nompelis at nobelware.com Cc: friends at talk.begriffs.com Subject: Re: Weds meeting > > The Hack Factory open house starts at 7 and I've been asked to wait outside until that time. So you might consider meeting elsewhere before 7. Oh! Thanks Dave. OK. IN that case I will skip it. Sorry guys. From nompelis at nobelware.com Thu Oct 10 14:30:19 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Thu, 10 Oct 2019 14:30:19 +0000 Subject: Weds meeting In-Reply-To: <9hrn96m9ldbrp3h6vte7sg94.1570665938215@nor.mn> References: <20191009182714.GA12618@nobelware.com> <9hrn96m9ldbrp3h6vte7sg94.1570665938215@nor.mn> Message-ID: <20191010143019.GA31041@nobelware.com> How did it go yesterday? From dave.bucklin at gmail.com Mon Oct 14 01:06:27 2019 From: dave.bucklin at gmail.com (Dave Bucklin) Date: Sun, 13 Oct 2019 20:06:27 -0500 Subject: Alternative Meetup Time Message-ID: <20191014010627.GA29808@19a6.tech> Hi folks. I'm interested in finding out if there's interest in setting an alternative meetup time. Wednesdays at the Hack Factory will still be available. To give more folks a chance to participate in person, I'm polling for interest in an alternative meetup time. Location TBD, but expect it to be in Minneapolis or a first-ring suburb; we can discuss that based on poll participation. This Doodle link is for a *representative* week. We can also discuss whether this will be weekly or alternate (e.g. first and third weeks of the month) or monthly. https://doodle.com/poll/3my9x23uayh8yxvw Thanks! From joe at begriffs.com Wed Oct 16 01:30:51 2019 From: joe at begriffs.com (Joe Nelson) Date: Tue, 15 Oct 2019 20:30:51 -0500 Subject: Idea: afterhours coworking Message-ID: <20191016013051.oxpwdne6im6dxwko@begriffs.com> In the latest installment of my brainstorming, I bring you another hot take. There are a lot of coworking spaces in the twin cities, Fueled Collective, WeWork, Industrious, Flock, Novel. However their vibe is businessy. People go there to grind away on their ecommerce website consulting or whatever then pack up around 5. So here's the idea: afterhours coworking for people building open source and side projects. If we were to solicit tech workers from the entire metro, regardless of programming language or industry, would we have the critical mass so that there would be activity every night at a place like this? I'm imagining just dropping in some random night and being able to get immersed in ideas and energy, hearing people talking, working on code together, hacking a music synth, dreaming things up. It'd be a different world in there. This came to mind because my employer does dollar matching for charitable giving, and I saw that TC Maker is an approved 503(c) on their list. I was thinking how much would it cost to outfit the hack factory classroom as a more comfortable lounge? Get a fridge in there, and a water filter that doesn't look like Chernobyl. Get some monitors, standing desks, and dual keyboards. Some warmer more relaxing track lights. The $55/mo regular membership might be too steep to get the growth we want to see, but perhaps they'd be open to offering a special membership level that is just for software people and excludes power tool use. Perhaps if we had a petition with enough interested would-be members then we'd have more clout proposing the idea to the hack factory. From joe at begriffs.com Wed Oct 16 16:33:39 2019 From: joe at begriffs.com (Joe Nelson) Date: Wed, 16 Oct 2019 11:33:39 -0500 Subject: Alternative Meetup Time In-Reply-To: <20191014010627.GA29808@19a6.tech> References: <20191014010627.GA29808@19a6.tech> Message-ID: <20191016163339.xmvrpkvt6mus2els@begriffs.com> > To give more folks a chance to participate in person, I'm > polling for interest in an alternative meetup time. > ... > https://doodle.com/poll/3my9x23uayh8yxvw Thanks for doing this Dave, that's a great idea. To all the lurkers out there, please do jump in and mark your preferences. We've got four responses so far, vs 45 people on the list. From nompelis at nobelware.com Thu Oct 17 12:20:53 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Thu, 17 Oct 2019 12:20:53 +0000 Subject: Idea: afterhours coworking In-Reply-To: <20191016013051.oxpwdne6im6dxwko@begriffs.com> References: <20191016013051.oxpwdne6im6dxwko@begriffs.com> Message-ID: <20191017122053.GA2950@nobelware.com> (Long email warning) Wow! Joe, you should really go into business management. I say this because you keep thinking on all fronts, from technical to social, and with a mind on productivity. I also sense a restlessness and desire to produce software a product -- I will return on this in a bit. I will tell you what I think. From my observations at the Hack Factory, it is not the kind of place where software people hang out. It certainly can be, but the vibe is different. (And I really like the hardware-oriented people and the vibe they have!) Also, from having observed how people worked back in the USENET days, it seems to me that software-type hacking and group-development has always had scarse and almost rogue timelines. People got together and hacked on a particular project for 1-2 days, or a few hours (kind of how we do it), and then they went home to what they were doing and stayed there for 2-3 months. Then, there are the software projects that occupy people's leisure time and have no physical space that they occupy other than people's home office. And in the post high-speed internet boom, group programming with skype/murmur and shared screens is just very appealing. (We do that!) I am not advocating not getting together. I think it is of great value and I always have a great time. I am advocating against eliminating the "rogue" element or adding more structure. I prefer less structure and mroe flexibility. It is laready hard for me, personally, to commit to working on projects, let alone put the effort to regularly go to a physical location to do it. (The data is there from my participation.) I would like to hear others' opinions on this, just to get an idea of what people are thinking and to add to the discussion. Returning to the desire to create, I do have some thoughts. I knwo that we all have different projets in mind, and there is overlap across them, etc. Several times we listed what intersts us and what we are working on. And at times, we worked on it together. (That's great!) But I really believe that it is not so much about getting critical mass to get an open source software product made. I think it is more about selfishness... One person really wants to see a thing made, so they work on it. If it is interesting, more people pick up on it and help, or spawn a different branch and go on their own, etc. So, find your own inspiration and form yoru self-motivation and go for it. Keep sharing what you do, what you find, etc, etc. Write your weblogs, write articles on 2600, or whatever. If something really spectacular comes up, we will follow. I regularly check your websites... Dave and Joe have been slaking with their weblogs! I will finish with a list of topics that interest me: - 3D graphcis and VR - music synthesizers - math - cryptography and cryptanalysis - algorithmic trading. - cryptocurrencies (as of lately) From nompelis at nobelware.com Thu Oct 17 12:28:01 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Thu, 17 Oct 2019 12:28:01 +0000 Subject: our VPSs Message-ID: <20191017122801.GA4009@nobelware.com> I have been doing some hacking on our OpenBSD VPS primarily. And I noticed that the UUCP node is still up. I say we kill the UUCP node. Also, Dave I have $50 for you to cover costs forthe VPSs. Next HackFactory meetup I show up and meet and transfer US currency bills to you. Let's shoot for next week. On the OpenBSD, I have been doing a lot of testing, mostly of my software. It has helped me enormously to maintain portability of my codes that do networking (OpenSSL based and straight TCP/UDP sockets). I strongly recommend that those who have not provided SSH keys to get accounts on the VPS get the keys to Joe/Dave. The system is damn solid too. I have built the environment modules for the system, but I have also ran into some trouble. Perhaps somebody can help me fix this next Wednesday. From joe at begriffs.com Thu Oct 17 18:35:18 2019 From: joe at begriffs.com (Joe Nelson) Date: Thu, 17 Oct 2019 13:35:18 -0500 Subject: IoT hackday Message-ID: <20191017183518.m2q7ukzluxt2dlcl@begriffs.com> Hey all, Daniel Feldman (CC'd) sent me a link for an upcoming hack day. Passing it along for anyone interested. Also CC'ing Chris and Mohit who have built some amazing hardware. https://iothackday2019.devpost.com/ It's kind of late notice, but looks like there is a meet and greet *tonight* for people to discuss projects and form teams. https://www.meetup.com/iotfuse/events/264601870/ From nompelis at nobelware.com Thu Oct 17 19:21:27 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Thu, 17 Oct 2019 19:21:27 +0000 Subject: IoT hackday In-Reply-To: <20191017183518.m2q7ukzluxt2dlcl@begriffs.com> References: <20191017183518.m2q7ukzluxt2dlcl@begriffs.com> Message-ID: <20191017192127.GA21310@nobelware.com> I have been to a couple of those, and they were fun. I will attempt to participate the day of. Highly encouraged to attend, you are. It is a great way to meet people and to get fresh ideas. Thanks for sending guys. From salo at saloits.com Thu Oct 17 21:04:31 2019 From: salo at saloits.com (Timothy J. Salo) Date: Thu, 17 Oct 2019 16:04:31 -0500 Subject: IoT hackday In-Reply-To: <20191017183518.m2q7ukzluxt2dlcl@begriffs.com> References: <20191017183518.m2q7ukzluxt2dlcl@begriffs.com> Message-ID: On 10/17/2019 1:35 PM, Joe Nelson wrote: > Hey all, Daniel Feldman (CC'd) sent me a link for an upcoming hack day. > Passing it along for anyone interested. Also CC'ing Chris and Mohit who > have built some amazing hardware. Although I'm sure he can speak for himself, I was under the impression (i.e., Instagram) that Mohit Bhoite is now in San Francisco. Of course, it is always nice to have support from Particle, in any form. Maybe, there is someone else from Particle locally who might participate. By the way, has anyone heard if Particle has summer internships for undergraduates? (No, I'm not the undergraduate in question...) -tjs From nompelis at nobelware.com Thu Oct 17 22:10:37 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Thu, 17 Oct 2019 17:10:37 -0500 Subject: IoT hackday In-Reply-To: References: <20191017183518.m2q7ukzluxt2dlcl@begriffs.com> Message-ID: Mohit is local and works for Particle. He used to be on our list. From joe at begriffs.com Fri Oct 18 02:56:20 2019 From: joe at begriffs.com (Joe Nelson) Date: Thu, 17 Oct 2019 21:56:20 -0500 Subject: Huh, I can't beat the Haskell performance in C Message-ID: <20191018025620.cahtjn7oqk7go3mm@begriffs.com> I saw this article about writing a fast version of the "wc" utility in Haskell and thought pfff, I can probably easily beat it with C: https://chrispenner.ca/posts/wc So I made two versions: simple and multi-threaded. But what the hell, the Haskell version is still beating me! https://github.com/begriffs/wc Can anyone spot inefficiencies in what I wrote? From eric at faehnri.ch Fri Oct 18 12:11:29 2019 From: eric at faehnri.ch (Eric Faehnrich) Date: Fri, 18 Oct 2019 08:11:29 -0400 Subject: Huh, I can't beat the Haskell performance in C In-Reply-To: <20191018025620.cahtjn7oqk7go3mm@begriffs.com> References: <20191018025620.cahtjn7oqk7go3mm@begriffs.com> Message-ID: I haven't dug into any implementation here, but I notice from the article that memory usage is always higher for the Haskell version, the trade off might be there. I'd want to see more analysis than just wall time, but also paging and disk IO (the article says someone mentioned disk caching.) -Eric From chuck at cpjj.net Fri Oct 18 12:37:32 2019 From: chuck at cpjj.net (Chuck Jungmann) Date: Fri, 18 Oct 2019 07:37:32 -0500 Subject: Huh, I can't beat the Haskell performance in C In-Reply-To: <20191018025620.cahtjn7oqk7go3mm@begriffs.com> References: <20191018025620.cahtjn7oqk7go3mm@begriffs.com> Message-ID: <2e8fe761-704d-1611-74ed-e248389f4561@cpjj.net> I saw your version on my GitHub feed and left a comment, but I'll relate part of it. I noticed that your for-loop tests for both the byte in range AND for eof.? It seems that when you set the thread ranges, you have already proven the range is contained in the file and the eof test is unnecessary. I left out another suggestion because I'm not sure it matters because I don't know much about C internals.? However, if C has to repeatedly calculate the addresses of the count and chunk struct members when accessing then, you could make pointers to these variables outside the loop and use the pointers in the loop to avoid the address math.? It's a trivial optimization that the compiler probably makes without any optimization flags, but it came to my mind. That's my two cent's worth... On 10/17/19 9:56 PM, Joe Nelson wrote: > I saw this article about writing a fast version of the "wc" utility in > Haskell and thought pfff, I can probably easily beat it with C: > https://chrispenner.ca/posts/wc > > So I made two versions: simple and multi-threaded. But what the hell, > the Haskell version is still beating me! > https://github.com/begriffs/wc > > Can anyone spot inefficiencies in what I wrote? > From nompelis at nobelware.com Fri Oct 18 14:23:44 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Fri, 18 Oct 2019 14:23:44 +0000 Subject: Huh, I can't beat the Haskell performance in C In-Reply-To: <2e8fe761-704d-1611-74ed-e248389f4561@cpjj.net> References: <20191018025620.cahtjn7oqk7go3mm@begriffs.com> <2e8fe761-704d-1611-74ed-e248389f4561@cpjj.net> Message-ID: <20191018142344.GA2363@nobelware.com> This is interesting. I did not see Haskell beat wc from GNU coreutils in the glance that I gave to this article. A few tricks, like streaming I/O, etc, play a role. I would run such tests over a RAM-disk, not a drive. Here is the GNU source, which, I noticed, makes use of "break" statements within loops, but is similar to Joe's: https://www.gnu.org/software/cflow/manual/html_node/Source-of-wc-command.html The total number of really functional lines-of-code of the GNU source is very small, like Joe's. The "reference" code this guy gives is much heavier. (Looks like it came from BSD.) I like his conclusion, that getting performance close to raw C with all the benefits of a fully type-checked programming language is great. I would add that for a language with garbage collection, it is even more impressive. From nompelis at nobelware.com Fri Oct 18 15:58:56 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Fri, 18 Oct 2019 15:58:56 +0000 Subject: Huh, I can't beat the Haskell performance in C In-Reply-To: <5141792E-5CA6-4C26-96F6-01DE0C218775@netjack.com> References: <20191018025620.cahtjn7oqk7go3mm@begriffs.com> <2e8fe761-704d-1611-74ed-e248389f4561@cpjj.net> <20191018142344.GA2363@nobelware.com> <5141792E-5CA6-4C26-96F6-01DE0C218775@netjack.com> Message-ID: <20191018155856.GA6813@nobelware.com> I am quoting Andrew's message below because it did not get to the list, > > I just sent a pull request. > I modified simple.c to use a 1 MB buffer ? HUGE speed up. 50X with my single test data file. > > https://github.com/begriffs/wc/pull/1 > He also sent a subsequent message, which also did not get to the list and is as follows: > > even decreasing to 1 KB buffer (instead of 1 MB) is a gigantic = > speed-up, a bit over 20X. > > Hmm, on the last message I just got an auto-reply that said "The = > message's content type was not explicitly allowed=E2=80=9D. Does that = > mean it=E2=80=99s going nowhere? > > Thanks! > I checked the archive and it did not list his messages: https://talk.begriffs.com/pipermail/friends/2019-October/thread.html From joe at begriffs.com Sat Oct 19 06:59:35 2019 From: joe at begriffs.com (Joe Nelson) Date: Sat, 19 Oct 2019 01:59:35 -0500 Subject: Huh, I can't beat the Haskell performance in C In-Reply-To: <20191018025620.cahtjn7oqk7go3mm@begriffs.com> References: <20191018025620.cahtjn7oqk7go3mm@begriffs.com> Message-ID: <20191019065935.GB34849@begriffs.com> I asked: > Can anyone spot inefficiencies in what I wrote? Andrew Benson sent me a pull request where he replaced fgetc() with read() and used a 1MB buffer. This improved speed significantly! I made the same type of change in the threaded version and now I think it goes faster than the Haskell one. I wrote Chris Penner, the Haskell author, to have him try it on his same hardware with his same data files to confirm. Try pulling the latest version and see how fast it goes for you. Perhaps there is even more optimization we can do. From salo at saloits.com Sat Oct 19 08:19:34 2019 From: salo at saloits.com (Timothy J. Salo) Date: Sat, 19 Oct 2019 03:19:34 -0500 Subject: Huh, I can't beat the Haskell performance in C In-Reply-To: <20191018025620.cahtjn7oqk7go3mm@begriffs.com> References: <20191018025620.cahtjn7oqk7go3mm@begriffs.com> Message-ID: <935319e6-dc22-aa6c-4c0a-e36f89628f25@saloits.com> On 10/17/2019 9:56 PM, Joe Nelson wrote: > I saw this article about writing a fast version of the "wc" utility in > Haskell and thought pfff, I can probably easily beat it with C: > https://chrispenner.ca/posts/wc > > So I made two versions: simple and multi-threaded. But what the hell, > the Haskell version is still beating me! > https://github.com/begriffs/wc > > Can anyone spot inefficiencies in what I wrote? Well, there are an awful lot of things that can affect program performance. The language [specification] is just one of them. Some languages do specify more work than others, such as Java's requirement for bounds checking for array references, versus C, which doesn't have such a requirement, (it's better to be fast, than to be safe and correct, I guess). Compilers make a big difference. Some compilers simply do a better job than others (e.g., some compiles generate more efficient object code). Of course, some compilers allow you to specify the desired optimization level, so asking the compiler for a higher optimization level may generate faster code. Is your version of Haskell a front end for a C compiler? If so, I would look at the C code generated by the Haskell front end. If your Haskell front end calls gcc, I would check what parameters it passes to gcc, specifically if Haskell asks gcc for a higher level of optimization. The language's runtime environment makes a difference. A language such as Java that runs in a language-specific virtual machine, might be expected to run more slowly than a language that runs in a native process, (ignoring the plethora of other factors that affect program speed). Even for programming languages that run in native processes, they may use different runtime libraries. Of course, all this stuff is probably open source, so you could, for example, compare the glibc implementation with whatever the Haskell equivalent is (which might actually be libc, e.g., Google [Haskell libc]). (Did I mention that performance analysis quickly becomes a black hole?) I notice that your C version uses unsigned long long integers. Multi-precision arithmetic can be pretty expensive on some hardware architectures. I would try running your C code with ints, long ints, unsigned long ints, etc., and compare the performance. I suspect that unsigned integer arithmetic is more expensive, because I think code must be generated to ensure unsigned arithmetic on hardware that doesn't natively support it. How big in an int in the Haskell that your are using? (Did I remember to mention that different languages might run relatively faster or slower on different hardware architectures. For example, if your program has to emulate floating-point operations because the hardware doesn't have a floating point unit, now you are comparing the quality of floating point emulation across languages.) Before drawing any broad conclusions, I would look at the assembly language generated by each language/compiler. Or, at least the C code, if it is available for both languages. I/O can make a big difference. The quality of the runtime library may differ between languages. And, you have other issues, as has been mentioned and which might be not readily visible, such as I/O buffer sizes. The interaction of the program with the memory system can make a difference. I think that I can generate a C program that will exhibit several orders-of-magnitude difference in performance, depending on how arrays are laid out in memory and how the arrays are accessed. Programs can exhibit different performance depending on what other programs are running or how much I/O is going on. If a lot of other programs are running (sucking up memory), then your program might page a lot more (depending on its structure). So, to more directly compare language performance, I might avoid I/O. And, I might look at the assembly language generated by each compiler, and see if I could avoid constructs that one language or the other finds particularly expensive. How are you measuring speed? Elapsed wall clock time? CPU time? -tjs From pmooney at pfmooney.com Sat Oct 19 13:58:19 2019 From: pmooney at pfmooney.com (Patrick Mooney) Date: Sat, 19 Oct 2019 08:58:19 -0500 Subject: Huh, I can't beat the Haskell performance in C In-Reply-To: <20191019065935.GB34849@begriffs.com> References: <20191018025620.cahtjn7oqk7go3mm@begriffs.com> <20191019065935.GB34849@begriffs.com> Message-ID: <1781bf0e-1eb6-4788-889e-9cf50d73b919@www.fastmail.com> > Andrew Benson sent me a pull request where he replaced fgetc() with > read() and used a 1MB buffer. This improved speed significantly! I made > the same type of change in the threaded version and now I think it goes > faster than the Haskell one. I wrote Chris Penner, the Haskell author, > to have him try it on his same hardware with his same data files to > confirm. Buffering is definitely the kind of low-hanging fruit to go after in a case like this. Without going in deep on compiler optimizations or micro architecture details, you still want to drive down the cost per byte of processing overhead. An implementation using getc() will be at the mercy of libc's buffered IO system, which probably defaults to a buffer size between 256 and 4096, depending on the specific implementation. The cost of a syscall to fill that buffer every time it is emptied vastly outweighs the logic to process its contents like wc(1). Threading access to getc() could induce even more per-byte cost, since libc will need to synchronize access to that buffer with an exclusion primitive. (Whereas many implementations will skip such checks when they know the calling program is single-threaded.) A large read(3)-filled buffer will do a good job of amortizing syscall costs across the rest of the relatively simple processing. A smart VM system in the host OS may even drive down your allocation costs when the file size falls far short of the buffer, since it could zero-fill-on-demand those pages as they are written to on the first read() call. -- Patrick Mooney From drewbenson at netjack.com Sat Oct 19 14:19:32 2019 From: drewbenson at netjack.com (Andrew Benson) Date: Sat, 19 Oct 2019 09:19:32 -0500 Subject: Huh, I can't beat the Haskell performance in C In-Reply-To: <1781bf0e-1eb6-4788-889e-9cf50d73b919@www.fastmail.com> References: <1781bf0e-1eb6-4788-889e-9cf50d73b919@www.fastmail.com> Message-ID: <2FF19050-D592-45CC-8ED4-E080DC654BF8@netjack.com> Right, that was my thought with the buffering. Running on just my Mac with a single 100,000 line test file (of SQL commands), the 1MB buffer was about 50X faster, and a 1 KB buffer was still 20X faster. I did test a 1 byte buffer also, which went on much longer than the ?simple.c? implementation, so I?m guessing it?s probably 256 byte or similar. In these samples, the I/O was reading from stdin. If reading from a file in any system which supports it, I would expect to see better performance yet using memory-mapped I/O ? ie, memory mapping the entire file into the process?s address space with the VM system, looping through the mapped memory, and letting page faults do the dirty work. At least on UNIX/Linux systems, I would expect that to have good performance, but I?d be curious to see if it does better than larger buffer sizes of direct file I/O. > On Oct 19, 2019, at 8:59 AM, Patrick Mooney wrote: > ? > > Andrew Benson sent me a pull request where he replaced fgetc() with > read() and used a 1MB buffer. This improved speed significantly! I made > the same type of change in the threaded version and now I think it goes > faster than the Haskell one. I wrote Chris Penner, the Haskell author, > to have him try it on his same hardware with his same data files to > confirm. Buffering is definitely the kind of low-hanging fruit to go after in a case like this. Without going in deep on compiler optimizations or micro architecture details, you still want to drive down the cost per byte of processing overhead. An implementation using getc() will be at the mercy of libc's buffered IO system, which probably defaults to a buffer size between 256 and 4096, depending on the specific implementation. The cost of a syscall to fill that buffer every time it is emptied vastly outweighs the logic to process its contents like wc(1). Threading access to getc() could induce even more per-byte cost, since libc will need to synchronize access to that buffer with an exclusion primitive. (Whereas many implementations will skip such checks when they know the calling program is single-threaded.) A large read(3)-filled buffer will do a good job of amortizing syscall costs across the rest of the relatively simple processing. A smart VM system in the host OS may even drive down your allocation costs when the file size falls far short of the buffer, since it could zero-fill-on-demand those pages as they are written to on the first read() call. -- Patrick Mooney From kyle.marek.spartz at gmail.com Sat Oct 19 17:41:58 2019 From: kyle.marek.spartz at gmail.com (Kyle Marek-Spartz) Date: Sat, 19 Oct 2019 12:41:58 -0500 Subject: Huh, I can't beat the Haskell performance in C In-Reply-To: <20191018025620.cahtjn7oqk7go3mm@begriffs.com> References: <20191018025620.cahtjn7oqk7go3mm@begriffs.com> Message-ID: <961CA0BC-0AD6-4681-B67E-5B1796765A4C@gmail.com> > On Oct 17, 2019, at 9:56 PM, Joe Nelson wrote: > > I saw this article about writing a fast version of the "wc" utility in > Haskell and thought pfff, I can probably easily beat it with C: > https://chrispenner.ca/posts/wc > > So I made two versions: simple and multi-threaded. But what the hell, > the Haskell version is still beating me! > https://github.com/begriffs/wc > > Can anyone spot inefficiencies in what I wrote? Haskell?s threads are user space threads, not kernel threads. You could try something like http://libmill.org Another thought would be to emit https://en.m.wikipedia.org/wiki/C-- or LLVM IR from the Haskell compiler, then port that, then make it idiomatic. From dave.bucklin at gmail.com Sat Oct 19 23:23:25 2019 From: dave.bucklin at gmail.com (Dave Bucklin) Date: Sat, 19 Oct 2019 18:23:25 -0500 Subject: our VPSs In-Reply-To: <20191017122801.GA4009@nobelware.com> References: <20191017122801.GA4009@nobelware.com> Message-ID: <20191019232325.GA601@19a6.tech> On Thu, Oct 17, 2019 at 12:28:01PM +0000, Ioannis Nompelis wrote: > I say we kill the UUCP node. Axed. Also, we could stand to upgrade OpenSBD to 6.6. From zach at nor.mn Sat Oct 19 23:29:50 2019 From: zach at nor.mn (Zach Norman) Date: Sat, 19 Oct 2019 18:29:50 -0500 Subject: our VPSs In-Reply-To: <20191019232325.GA601@19a6.tech> Message-ID: Why not just use mullvad? They probably have better security then any of us can setup in our freetime. They allow cash for payment if anonymity is a concern. https://mullvad.net/en/ Zach normnco automata ? Original Message ? From: dave.bucklin at gmail.com Sent: October 19, 2019 18:23 To: nompelis at nobelware.com Cc: friends at talk.begriffs.com Subject: Re: our VPSs On Thu, Oct 17, 2019 at 12:28:01PM +0000, Ioannis Nompelis wrote: > I say we kill the UUCP node. Axed. Also, we could stand to upgrade OpenSBD to 6.6. From dave.bucklin at gmail.com Sun Oct 20 00:07:55 2019 From: dave.bucklin at gmail.com (Dave Bucklin) Date: Sat, 19 Oct 2019 19:07:55 -0500 Subject: Alternative Meetup Time In-Reply-To: <20191016163339.xmvrpkvt6mus2els@begriffs.com> References: <20191014010627.GA29808@19a6.tech> <20191016163339.xmvrpkvt6mus2els@begriffs.com> Message-ID: <20191020000755.GA1532@19a6.tech> The votes are in. Here's where things stand. Sunday 2-4 4 votes Monday 6-8 5 votes Tuesday 6-8 5 votes Thursday 6-8 6 votes Thursday 8-10 4 votes Saturday 2-4 4 votes Saturday 4-6 4 votes It looks like Thursday is the most-favored alternative. Next, I think we need to decide on a place. When I opened the poll, I said Minneapolis or first-ring suburb and I'd like to stick to that. So, get out your map and compass. I'd like to hear up to three nominations from those of you wanting to meet up Thursday evening. We'll vote on that and then set the date. Thanks! From joe at begriffs.com Sun Oct 20 04:30:56 2019 From: joe at begriffs.com (Joe Nelson) Date: Sat, 19 Oct 2019 23:30:56 -0500 Subject: our VPSs In-Reply-To: <20191019232325.GA601@19a6.tech> References: <20191017122801.GA4009@nobelware.com> <20191019232325.GA601@19a6.tech> Message-ID: <20191020043056.vnfpmcvbutvqq6lh@begriffs.com> Dave Bucklin wrote: > Also, we could stand to upgrade OpenSBD to 6.6. Yeah 6.4 is getting older now. I noticed the mirror we were using for obsd packages no longer has 6.4 packages, it keeps only the two most recent versions. I found an alternative, https://mirror.esc7.net/pub/OpenBSD, in Dallas which continues to offer 6.4 packages. I updated /etc/installurl today to point at it. That's what pkg_add consults. Another interesting use for our VPS is to host CalDAV. This would allow us to link standard calendar apps on computers or phones to a our server. Our group events could then show up on the calendar, and I think people could RSVP through those apps rather than replying on an email chain to do it. RSVPs via email (or even meetup.com) feels a little clunky compared to a real calendar app. I installed radicale [0] on the VPS and started its server running. You can learn more about the installation in /usr/local/share/doc/pkg-readmes/radicale. It looks like the server supports having user accounts, but may also support free-for-all editing which would be convenient since we have security by obscurity. 0: https://radicale.org From nompelis at nobelware.com Sun Oct 20 15:27:56 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Sun, 20 Oct 2019 15:27:56 +0000 Subject: our VPSs In-Reply-To: <20191020043056.vnfpmcvbutvqq6lh@begriffs.com> References: <20191017122801.GA4009@nobelware.com> <20191019232325.GA601@19a6.tech> <20191020043056.vnfpmcvbutvqq6lh@begriffs.com> Message-ID: <20191020152756.GA11035@nobelware.com> We have better security than just "via obscurity". (By the way, do not ever underestimate the power of obscurity in security; one little tweak and the cost of breaking security exponentiates.) I will look into the calendar app. And we should do and update to the VPS and see which of my software breaks... From nompelis at nobelware.com Sun Oct 20 15:29:27 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Sun, 20 Oct 2019 15:29:27 +0000 Subject: our VPSs In-Reply-To: References: <20191019232325.GA601@19a6.tech> Message-ID: <20191020152927.GB11035@nobelware.com> > Why not just use mullvad? They probably have better security then any of us can setup in our freetime. They allow cash for payment if anonymity is a concern. > > https://mullvad.net/en/ > Zach, I think we are pretty good with what we have. And few of us (mostly me) use this on a semi-regular basis. But it is great to know one can pay cash for network access. From zach at nor.mn Sun Oct 20 15:31:11 2019 From: zach at nor.mn (Zach Norman) Date: Sun, 20 Oct 2019 10:31:11 -0500 Subject: our VPSs In-Reply-To: <20191020152927.GB11035@nobelware.com> Message-ID: <84205et03d54cvi6bqb2lhks.1571585471185@nor.mn> Also I misread this thread subject -- vps not VPN. normnco automata ? Original Message ? From: nompelis at nobelware.com Sent: October 20, 2019 10:29 To: zach at nor.mn Reply-to: nompelis at nobelware.com Cc: dave.bucklin at gmail.com; friends at talk.begriffs.com Subject: Re: our VPSs > Why not just use mullvad? They probably have better security then any of us can setup in our freetime. They allow cash for payment if anonymity is a concern. > > https://mullvad.net/en/ > Zach, I think we are pretty good with what we have. And few of us (mostly me) use this on a semi-regular basis. But it is great to know one can pay cash for network access. From nompelis at nobelware.com Sun Oct 20 15:34:08 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Sun, 20 Oct 2019 15:34:08 +0000 Subject: our VPSs In-Reply-To: <84205et03d54cvi6bqb2lhks.1571585471185@nor.mn> References: <20191020152927.GB11035@nobelware.com> <84205et03d54cvi6bqb2lhks.1571585471185@nor.mn> Message-ID: <20191020153408.GC11035@nobelware.com> Speaking of VPN, has anyone tried that "ad hoc" user-network-space VPN for Linux that was based on some python and crap like that? I am always interested in VPNs that are hacks and require low privilege accounts. From joe at begriffs.com Sun Oct 20 23:30:00 2019 From: joe at begriffs.com (Joe Nelson) Date: Sun, 20 Oct 2019 18:30:00 -0500 Subject: Alternative Meetup Time In-Reply-To: <20191020000755.GA1532@19a6.tech> References: <20191014010627.GA29808@19a6.tech> <20191016163339.xmvrpkvt6mus2els@begriffs.com> <20191020000755.GA1532@19a6.tech> Message-ID: <20191020233000.GD34849@begriffs.com> Dave Bucklin wrote: > Thursday 6-8 6 votes > Thursday 8-10 4 votes > > It looks like Thursday is the most-favored alternative. Another benefit it's the only time Sam marked as available. :) Works for me. > Next, I think we need to decide on a place. When I opened the poll, I > said Minneapolis or first-ring suburb and I'd like to stick to that. So, > get out your map and compass. I'd like to hear up to three nominations > from those of you wanting to meet up Thursday evening. We'll vote on > that and then set the date. Here are four ideas: * WeWork Capella tower. They don't charge anything if we list the event on Meetup, although then we'd have to pay Meetup... The location is pretty good, a block from a train station. Parking may not be so great though, and the checkin process at desk in the lobby is more complicated than just walking in. * Silicon Prairie Portal & Exchange, 475 Cleveland Ave N. Right on the border between the cities, off a highway. Ioannis hooked us up with this location and maybe he could do so again. They had a bunch of chairs but not sure if they have tables for everyone. * Conference room at the Minneapolis central library. It has whiteboards and tables, and allows the hours 6-8. No food allowed though. https://hclib.libcal.com/space/34808 * Parkway pizza? From nompelis at nobelware.com Mon Oct 21 13:52:06 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Mon, 21 Oct 2019 13:52:06 +0000 Subject: Alternative Meetup Time In-Reply-To: <20191020233000.GD34849@begriffs.com> References: <20191014010627.GA29808@19a6.tech> <20191016163339.xmvrpkvt6mus2els@begriffs.com> <20191020000755.GA1532@19a6.tech> <20191020233000.GD34849@begriffs.com> Message-ID: <20191021135206.GA8167@nobelware.com> I like the idea of the central library for two reasons: 1. let's see how it is and use the money we pay, and 2. the enviornment may really make us focus on something good From joe at begriffs.com Mon Oct 21 14:48:21 2019 From: joe at begriffs.com (Joe Nelson) Date: Mon, 21 Oct 2019 09:48:21 -0500 Subject: Alternative Meetup Time In-Reply-To: <20191021135206.GA8167@nobelware.com> References: <20191014010627.GA29808@19a6.tech> <20191016163339.xmvrpkvt6mus2els@begriffs.com> <20191020000755.GA1532@19a6.tech> <20191020233000.GD34849@begriffs.com> <20191021135206.GA8167@nobelware.com> Message-ID: <158aab7a-174c-49ae-bbbc-af1b7625c88c@www.fastmail.com> > I like the idea of the central library I put in a request to book the room: N-402 - Minneapolis Central: 6:00pm - 9:00pm, Thursday, October 24 If we want to meet somewhere else that's fine, but wanted to move fast and snag the room this week just in case. From nompelis at nobelware.com Tue Oct 22 18:58:29 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Tue, 22 Oct 2019 18:58:29 +0000 Subject: Alternative Meetup Time In-Reply-To: <158aab7a-174c-49ae-bbbc-af1b7625c88c@www.fastmail.com> References: <20191014010627.GA29808@19a6.tech> <20191016163339.xmvrpkvt6mus2els@begriffs.com> <20191020000755.GA1532@19a6.tech> <20191020233000.GD34849@begriffs.com> <20191021135206.GA8167@nobelware.com> <158aab7a-174c-49ae-bbbc-af1b7625c88c@www.fastmail.com> Message-ID: <20191022185829.GA16750@nobelware.com> > I put in a request to book the room: > > N-402 - Minneapolis Central: 6:00pm - 9:00pm, Thursday, October 24 > Let us know if it is confirmed from the library. Otherwise, it is a "go". From samuel.stuewe at gmail.com Tue Oct 22 19:01:29 2019 From: samuel.stuewe at gmail.com (Sam Stuewe) Date: Tue, 22 Oct 2019 14:01:29 -0500 Subject: Alternative Meetup Time In-Reply-To: <20191022185829.GA16750@nobelware.com> References: <20191014010627.GA29808@19a6.tech> <20191016163339.xmvrpkvt6mus2els@begriffs.com> <20191020000755.GA1532@19a6.tech> <20191020233000.GD34849@begriffs.com> <20191021135206.GA8167@nobelware.com> <158aab7a-174c-49ae-bbbc-af1b7625c88c@www.fastmail.com> <20191022185829.GA16750@nobelware.com> Message-ID: Hey everyone, I am sorry this schedule focused on my personal availability, as my plans have just changed in an unavoidable way, and I won't be able to join on Thursday. I really hope that isn't a massive inconvenience for anyone. ): All the best, -Sam From pmooney at pfmooney.com Tue Oct 22 19:21:17 2019 From: pmooney at pfmooney.com (Patrick Mooney) Date: Tue, 22 Oct 2019 14:21:17 -0500 Subject: Huh, I can't beat the Haskell performance in C In-Reply-To: <2FF19050-D592-45CC-8ED4-E080DC654BF8@netjack.com> References: <1781bf0e-1eb6-4788-889e-9cf50d73b919@www.fastmail.com> <2FF19050-D592-45CC-8ED4-E080DC654BF8@netjack.com> Message-ID: On Sat, Oct 19, 2019, at 09:19, Andrew Benson wrote: > In these samples, the I/O was reading from stdin. If reading from a > file in any system which supports it, I would expect to see better > performance yet using memory-mapped I/O ? ie, memory mapping the entire > file into the process?s address space with the VM system, looping > through the mapped memory, and letting page faults do the dirty work. > At least on UNIX/Linux systems, I would expect that to have good > performance, but I?d be curious to see if it does better than larger > buffer sizes of direct file I/O. It's true that you'd likely skip one copy operation on the data (since the VM system could map the populated buffer pages directly into the process AS), but it might come out net-negative if the read-ahead behavior isn't aggressive enough. If it's not loading a bunch of populated pages into the AS at once, you might take a fault every time you cross page boundaries, incurring a cost similar to read(2) with a PAGESIZE buffer. Empirical testing would sort that out. -- Patrick Mooney From salo at saloits.com Tue Oct 22 21:52:24 2019 From: salo at saloits.com (Timothy J. Salo) Date: Tue, 22 Oct 2019 16:52:24 -0500 Subject: Huh, I can't beat the Haskell performance in C In-Reply-To: References: <1781bf0e-1eb6-4788-889e-9cf50d73b919@www.fastmail.com> <2FF19050-D592-45CC-8ED4-E080DC654BF8@netjack.com> Message-ID: On 10/22/2019 2:21 PM, Patrick Mooney wrote: > It's true that you'd likely skip one copy operation on the > data (since the VM system could map the populated buffer pages > directly into the process AS), but it might come out > net-negative if the read-ahead behavior isn't aggressive enough. > If it's not loading a bunch of populated pages into the AS > at once, you might take a fault every time you cross page > boundaries, incurring a cost similar to read(2) with a PAGESIZE > buffer. Empirical testing would sort that out. A handful of thoughts on performance testing follow: o Perhaps most relevant, these experiments are no longer comparing the performance of two programming languages, they are comparing approaches to implementing I/O. If you are interested in comparing programming languages, you might want to eliminate I/O as a [large] variable. Well, sometimes. Languages that require byte-by-byte processing of I/O impose a fair amount of overhead, or languages that run in a language-specific VM probably limit some of the I/O performance improvements you can make. o If you want to examine performance at this level, you probably want to think about time and clocks. - If you are running on an x86 machine, you should probably look at the Time Stamp Counter (TSC) register. The TSC is incremented every clock cycle. Therefore, it provides time resolution on the order of the clock cycle of your machine. You can read the TSC in C with a few inline assembly language instructions. There are some complications to using the TSC. The TSC registers are not synchronized across cores, so if your compare TSC values read from different cores, your result may be less accurate. As I recall, there is an instruction that reads both the TSC and the CPU number. Intel has some good documentation on the TSC and some sample code. - read "man 7 time" for a discussion of time and timers on Linux systems. - look at Linux high-resolution clocks. These clocks will, if I recall correctly, provide nanosecond resolution, although I don't know how often the clock is updated. (Actually, I think the value of the high-resolution clock is computed using the TSC and the system clock, but I could be wrong). - In short, with some effort, you can accurately measure really short time intervals. This can be useful. o If you are performing fine-grained time measurements, try to eliminate other activity on the machine. For example, a bunch of other processes that are demanding memory, may cause your process to page more. And, I/O (interrupts) unrelated to your process will slow your process down. Of course, some of this is beyond your control. For example, a ton of things happen on one-second boundaries that you can't do anything about. o Look at the /proc pseudo file system. /proc provides access to a number of kernel data structures. You can get data from your process' task control block, including information about paging, memory, I/O, and just about anything else in the TCB. o I think that disks and/or the kernel driver try to read more that one sector, unless an end-of-file in encountered. I think these extra sectors are buffered. They might even be buffered in the disk drive and/or in the kernel. o Linux I/O is really complicated. Disks are really, really, really slow, so there is a lot of code in Linux (and in the drive itself) to improve performance. o Looking at performance quickly drives one to think about low-level implementation details. This should be perceived as a good thing. -tjs From joe at begriffs.com Wed Oct 23 02:29:33 2019 From: joe at begriffs.com (Joe Nelson) Date: Tue, 22 Oct 2019 21:29:33 -0500 Subject: Meeting this Thursday Message-ID: <20191023022933.zgmam6i5uhqn3aor@begriffs.com> My booking has been accepted for the Minneapolis central library (300 Nicollet Mall). I got conference room N-402 from 6:00 - 9:00. No food allowed in there, so might want to venture out (before or after) for dinner. Lyon's Pub looks pretty nice and is close to the library... From joe at begriffs.com Wed Oct 23 04:20:40 2019 From: joe at begriffs.com (Joe Nelson) Date: Tue, 22 Oct 2019 23:20:40 -0500 Subject: Huh, I can't beat the Haskell performance in C In-Reply-To: References: <1781bf0e-1eb6-4788-889e-9cf50d73b919@www.fastmail.com> <2FF19050-D592-45CC-8ED4-E080DC654BF8@netjack.com> Message-ID: <20191023042040.GA40065@begriffs.com> > A handful of thoughts on performance testing follow: Thanks for the detailed email. > o Perhaps most relevant, these experiments are no longer comparing the > performance of two programming languages, they are comparing > approaches to implementing I/O. True, the whole comparison is kind of sloppy, including the original article. Haskell does have access to POSIX IO through the "unix" library distributed with GHC. The Haskell version could have used fdReadBuf rather than the fancy streaming stuff. https://downloads.haskell.org/~ghc/latest/docs/html/libraries/unix-2.7.2.2/System-Posix-IO.html > o If you want to examine performance at this level, you probably want > to think about time and clocks. > > ... > - look at Linux high-resolution clocks. Good idea, and looks like POSIX gives us clock_gettime(), where the resolution is system dependent and can be determined with clock_getres(). https://pubs.opengroup.org/onlinepubs/9699919799/functions/clock_getres.html I think it would be interesting to sprinkle my program with clock checks and add up the total time spent in IO, the counting logic, and thread calls. Perhaps we could do this at the Thursday hack night. > o If you are performing fine-grained time measurements, try to > eliminate other activity on the machine. Can anyone recommend a VPS to run the benchmark with minimal other activity? I wonder whether there are EC2 instances which are not shared with other tenants, where I can get the whole machine and have no other programs except the most basic OS services running. > o Look at the /proc pseudo file system. /proc provides access to > a number of kernel data structures. You can get data from your > process' task control block, including information about paging, > memory, I/O, and just about anything else in the TCB. What are some stats that would be relevant for the word count program? > > o I think that disks and/or the kernel driver try to read more that > one sector, unless an end-of-file in encountered. Is this sector concept specific to spinning drives only, or does the kernel use the same logic for SSDs? > o Looking at performance quickly drives one to think about low-level > implementation details. This should be perceived as a good thing. It's a good exercise to measure performance for wc since it does such a simple focused task. From prenticedarren at gmail.com Wed Oct 23 15:01:25 2019 From: prenticedarren at gmail.com (Darren Prentice) Date: Wed, 23 Oct 2019 10:01:25 -0500 Subject: Meeting this Thursday In-Reply-To: <20191023022933.zgmam6i5uhqn3aor@begriffs.com> References: <20191023022933.zgmam6i5uhqn3aor@begriffs.com> Message-ID: Hey all ? I?ll try and make it. Lyon?s sounds good. Downtown is nice for me, living in northeast. I?ve been swamped at work building this epic web gui I designed for routing multichannel Audio over IP. It uses svg Bezier curves to visualize ?virtual cable? connections, but one of the challenges is that when you add lots of these it gets very messy and hard to read. So something I might hack on tomorrow is writing an algorithm that does some automated #cableporn magic. From drewbenson at netjack.com Wed Oct 23 16:52:58 2019 From: drewbenson at netjack.com (Andrew Benson) Date: Wed, 23 Oct 2019 11:52:58 -0500 Subject: Meeting this Thursday In-Reply-To: References: <20191023022933.zgmam6i5uhqn3aor@begriffs.com> Message-ID: <337C04F8-BE6A-4381-B89C-E975B51FA6F0@netjack.com> That sounds pretty cool. :) > On Oct 23, 2019, at 10:01 AM, Darren Prentice wrote: > > I?ve been swamped at work building this epic web gui I designed for routing multichannel Audio over IP. It uses svg Bezier curves to visualize ?virtual cable? connections, but one of the challenges is that when you add lots of these it gets very messy and hard to read. So something I might hack on tomorrow is writing an algorithm that does some automated #cableporn magic. From zach at nor.mn Wed Oct 23 16:54:54 2019 From: zach at nor.mn (Zach Norman) Date: Wed, 23 Oct 2019 11:54:54 -0500 Subject: Meeting this Thursday In-Reply-To: <337C04F8-BE6A-4381-B89C-E975B51FA6F0@netjack.com> Message-ID: I've been looking to use jackd tools to do similar things in order to access USB enabled hardware instruments while on the road. Will be good to hear more about this. Zach normnco automata ? Original Message ? From: drewbenson at netjack.com Sent: October 23, 2019 11:53 To: prenticedarren at gmail.com Cc: friends at talk.begriffs.com Subject: Re: Meeting this Thursday That sounds pretty cool. :) > On Oct 23, 2019, at 10:01 AM, Darren Prentice wrote: > > I?ve been swamped at work building this epic web gui I designed for routing multichannel Audio over IP.? It uses svg Bezier curves to visualize ?virtual cable? connections, but one of the challenges is that when you add lots of these it gets very messy and hard to read. So something I might hack on tomorrow is writing an algorithm that does some automated #cableporn magic. From dave.bucklin at gmail.com Wed Oct 23 17:22:30 2019 From: dave.bucklin at gmail.com (Dave Bucklin) Date: Wed, 23 Oct 2019 12:22:30 -0500 Subject: Meeting this Thursday In-Reply-To: <20191023022933.zgmam6i5uhqn3aor@begriffs.com> References: <20191023022933.zgmam6i5uhqn3aor@begriffs.com> Message-ID: On October 22, 2019 9:29:33 PM CDT, Joe Nelson wrote: >My booking has been accepted for the Minneapolis central library (300 >Nicollet Mall). I got conference room N-402 from 6:00 - 9:00. > >No food allowed in there, so might want to venture out (before or >after) >for dinner. Lyon's Pub looks pretty nice and is close to the library... I'll be there. Pizza Luce is also just a couple blocks away. I used to spend a lot of time in Lyon's, back when you could smoke inside. From nompelis at nobelware.com Wed Oct 23 18:40:47 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Wed, 23 Oct 2019 18:40:47 +0000 Subject: Meeting this Thursday In-Reply-To: References: <337C04F8-BE6A-4381-B89C-E975B51FA6F0@netjack.com> Message-ID: <20191023184047.GA8925@nobelware.com> > > I've been looking to use jackd tools to do similar things in order to access USB enabled hardware instruments while on the road. Will be good to hear more about this. > I want to be part of this kind of orgy. From joe at begriffs.com Thu Oct 24 01:06:03 2019 From: joe at begriffs.com (Joe Nelson) Date: Wed, 23 Oct 2019 20:06:03 -0500 Subject: Meeting this Thursday In-Reply-To: <20191023184047.GA8925@nobelware.com> References: <337C04F8-BE6A-4381-B89C-E975B51FA6F0@netjack.com> <20191023184047.GA8925@nobelware.com> Message-ID: <20191024010603.ht5ybxmuwj2trlr5@begriffs.com> Ioannis Nompelis wrote: > I want to be part of this kind of orgy. Uhh, I think you accidentally replied to the wrong thread... ;) From joe at begriffs.com Thu Oct 24 02:52:23 2019 From: joe at begriffs.com (Joe Nelson) Date: Wed, 23 Oct 2019 21:52:23 -0500 Subject: Idea: afterhours coworking In-Reply-To: <20191017122053.GA2950@nobelware.com> References: <20191016013051.oxpwdne6im6dxwko@begriffs.com> <20191017122053.GA2950@nobelware.com> Message-ID: <20191024025223.GC79416@begriffs.com> > I will tell you what I think. From my observations at the Hack > Factory, it is not the kind of place where software people hang out. > It certainly can be, but the vibe is different. I know what you mean. I also don't sense much of a desire for change there. Hey good for them if they've got what they want. > But I really believe that it is not so much about getting critical > mass to get an open source software product made. I think it is more > about selfishness... One person really wants to see a thing made, so > they work on it. If it is interesting, more people pick up on it and > help, or spawn a different branch and go on their own, etc. So, find > your own inspiration and form yoru self-motivation and go for it. Yeah that's about right. > Keep sharing what you do, what you find, etc, etc. Write your weblogs, > write articles on 2600, or whatever. If something really spectacular > comes up, we will follow. Well said. I think I've been too anxious lately to collaborate on things. At the end of the day some people put in the work and others don't; just life as usual. I also feel like I've been saying a lot lately about things I'm doing or am going to do, and that it would be more productive to limit discussion to after the thing is done (or at least at v1). For some reason it kind of jinxes projects to talk about them early. From nompelis at nobelware.com Thu Oct 24 12:37:23 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Thu, 24 Oct 2019 07:37:23 -0500 Subject: Meeting this Thursday In-Reply-To: <20191024010603.ht5ybxmuwj2trlr5@begriffs.com> References: <337C04F8-BE6A-4381-B89C-E975B51FA6F0@netjack.com> <20191023184047.GA8925@nobelware.com> <20191024010603.ht5ybxmuwj2trlr5@begriffs.com> Message-ID: <946E4A8E-B15E-4667-9788-9A149D5C1381@nobelware.com> Instruments and ADWs is totally my jam! On Oct 23, 2019, at 8:06 PM, Joe Nelson wrote: > Ioannis Nompelis wrote: >> I want to be part of this kind of orgy. > > Uhh, I think you accidentally replied to the wrong thread... ;) From nompelis at nobelware.com Thu Oct 24 12:41:11 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Thu, 24 Oct 2019 07:41:11 -0500 Subject: Idea: afterhours coworking In-Reply-To: <20191024025223.GC79416@begriffs.com> References: <20191016013051.oxpwdne6im6dxwko@begriffs.com> <20191017122053.GA2950@nobelware.com> <20191024025223.GC79416@begriffs.com> Message-ID: <6C87990A-D33C-4A76-96CF-4C1727F8A328@nobelware.com> I think it does help to talk aboit. projects ahead of time. Torvalds got on USENET and asked for opinions on the OS he was going to make. I enjoy the discussion, althought it does not replace action! From joe at begriffs.com Thu Oct 24 22:19:49 2019 From: joe at begriffs.com (Joe Nelson) Date: Thu, 24 Oct 2019 17:19:49 -0500 Subject: Meeting this Thursday In-Reply-To: References: <20191023022933.zgmam6i5uhqn3aor@begriffs.com> Message-ID: <16c8bbca-65f2-49df-a7df-ca23456b55a1@www.fastmail.com> On Thu, Oct 24, 2019, at 5:04 PM, Christopher Pearson wrote: > I'll be at Lyon's in 5 minutes Cool, I'll head over there. From chris at mikk.net Thu Oct 24 23:12:59 2019 From: chris at mikk.net (Chris Mikkelson) Date: Thu, 24 Oct 2019 18:12:59 -0500 Subject: Meeting this Thursday In-Reply-To: <16c8bbca-65f2-49df-a7df-ca23456b55a1@www.fastmail.com> References: <20191023022933.zgmam6i5uhqn3aor@begriffs.com> <16c8bbca-65f2-49df-a7df-ca23456b55a1@www.fastmail.com> Message-ID: <29FA65C3-C19D-488D-93FA-CA6A1567E17E@mikk.net> I'm en route. Are folks still at Lyon's or have you moved to the library? On October 24, 2019 5:19:49 PM CDT, Joe Nelson wrote: >On Thu, Oct 24, 2019, at 5:04 PM, Christopher Pearson wrote: >> I'll be at Lyon's in 5 minutes > >Cool, I'll head over there. -- Sent from my Android device with K-9 Mail. Please excuse my brevity. From zach at nor.mn Thu Oct 24 23:16:32 2019 From: zach at nor.mn (Zach Norman) Date: Thu, 24 Oct 2019 18:16:32 -0500 Subject: Meeting this Thursday In-Reply-To: <29FA65C3-C19D-488D-93FA-CA6A1567E17E@mikk.net> Message-ID: Three of us are here at the library now Zach normnco automata ? Original Message ? From: chris at mikk.net Sent: October 24, 2019 18:13 To: friends at talk.begriffs.com; joe at begriffs.com; kermit4 at gmail.com; dave.bucklin at gmail.com Cc: friends at talk.begriffs.com Subject: Re: Meeting this Thursday I'm en route. Are folks still at Lyon's or have you moved to the library? On October 24, 2019 5:19:49 PM CDT, Joe Nelson wrote: >On Thu, Oct 24, 2019, at 5:04 PM, Christopher Pearson wrote: >> I'll be at Lyon's in 5 minutes > >Cool, I'll head over there. -- Sent from my Android device with K-9 Mail. Please excuse my brevity. From nompelis at nobelware.com Fri Oct 25 14:27:38 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Fri, 25 Oct 2019 14:27:38 +0000 Subject: Meeting this Thursday In-Reply-To: <20191023022933.zgmam6i5uhqn3aor@begriffs.com> References: <20191023022933.zgmam6i5uhqn3aor@begriffs.com> Message-ID: <20191025142738.GA17650@nobelware.com> Great to see everyone. There was a good number of us there. I did not "hack" on anything, but lots of technical chat was made (ALSA, JACK/jackd, Pulseaudio). I liked how the hacker in all of us came out when we were trying to get the RasPi on a network! I ended up chatting with Chris P. mostly, and it was about the state of the art supercomputer architectures with accelerators. We spent time looking at the Volta V100 based GPU systems that nvidia ships, which now include more elaborate hardware components, like the nvlink2 intra-node interconnect. Chris did not know that CUDA is not necessary to program on accelerators, and that OpenMP (4.5 and up) has support for this ("TARGET programming"). It still amazes me how much Chris knows about hardware and software in general for somebody who is not in the supercomputing industry. If I had a company that was involved in supercomputing --like some people I know have-- I would ask Chris to work alongside me. What were you guys working on that involved writing on the white board? We should do this more often. From chris at mikk.net Fri Oct 25 17:49:16 2019 From: chris at mikk.net (Chris Mikkelson) Date: Fri, 25 Oct 2019 12:49:16 -0500 Subject: Meeting this Thursday In-Reply-To: <20191025142738.GA17650@nobelware.com> References: <20191023022933.zgmam6i5uhqn3aor@begriffs.com> <20191025142738.GA17650@nobelware.com> Message-ID: <20191025174916.GA83588@mikk.net> On Fri, Oct 25, 2019 at 02:27:38PM +0000, Ioannis Nompelis wrote: > Great to see everyone. There was a good number of us there. I did not > "hack" on anything, but lots of technical chat was made (ALSA, JACK/jackd, > Pulseaudio). +1 As a remote worker, it's great to have a like-minded technical cohort to sit around and shoot the bits with! > What were you guys working on that involved writing on the white board? That was me and Joe chatting about memory barriers, the limitations of atomic operations, and other low-level topics in concurrent programming. The context / prompt was the following snippet of code: https://github.com/farsightsec/libmy/blob/master/my_queue_mb.c which implements a lock-free SPSC[1] queue. I've been making use of the above in some multithreaded projects of mine, and have also inherited maintenance of the code, making it useful to understand how it works :) -- Chris Mikkelson | Einstein himself said that God doesn't roll dice. But chris at mikk.net | he was wrong. And in fact, anyone who has played role- | playing games knows that God probably had to roll quite | a few dice to come up with a character like Einstein. | -- Larry Wall [1] SPSC = "Single Producer, Single Consumer" i.e. one thread writing at a time, one thread reading at a time. This reduces the need for synchronization -- the above implementation does not require mutexes or even atomic operations. From salo at saloits.com Fri Oct 25 20:27:27 2019 From: salo at saloits.com (Timothy J. Salo) Date: Fri, 25 Oct 2019 15:27:27 -0500 Subject: Huh, I can't beat the Haskell performance in C In-Reply-To: <20191023042040.GA40065@begriffs.com> References: <1781bf0e-1eb6-4788-889e-9cf50d73b919@www.fastmail.com> <2FF19050-D592-45CC-8ED4-E080DC654BF8@netjack.com> <20191023042040.GA40065@begriffs.com> Message-ID: <7d529522-0ce1-a746-c989-88c2e4b01401@saloits.com> On 10/22/2019 11:20 PM, Joe Nelson wrote: >> o I think that disks and/or the kernel driver try to read more that >> one sector, unless an end-of-file in encountered. > > Is this sector concept specific to spinning drives only, or does the kernel use > the same logic for SSDs? I some SSDs (such as, SSDs with SATA interfaces) are designed to look like hard drives, (both logically and physically), except faster. That is, these SSDs are advertised as drop-in replacements. I'm not sure what other SSD interfaces look like, (e.g., PCIe), but I assume that the look pretty much like hard drives. So, I think that SSDs are pretty much invisible/transparent to Linux, except perhaps at the lowest-level driver. Even there, the logical interface to the SSD looks a lot like hard drives (at least for SATA devices). I assume that the kernel would fall apart if you tried to use a sector size other than 512. I'm pretty sure that the kernel file system code (for random-access, block-mode devices) is all based on sectors (of 512 bytes). There are a few important exception to the notion that SSDs look exactly like hard drives. For example, flash memory can wear out after a certain number of writes. Specifically, flash memories are good for maybe 100,000 write cycles. Now, I think there are several recovery mechanisms intended to hide failing flash memories (I think on both the chip level, and certainly in the SSD controller), but still wearing out flash memory by repeatedly writing to the same sector is something to try to avoid. There are several file systems designed for flash memory (SSDs). One important motivation for these flash file systems is to distribute wear throughout the SSD, to avoid prematurely wearing out particular locations of the flash. Of course, there is a Wiki article on this: Be sure to also read the Wiki articles on wear leveling and write amplification. It is useful to remember that mass storage devices are very intelligent devices, in the sense that they have CPUs, a ton of code and quite a bit of memory (used for buffering). One way to get a sense of this is to read your hard drive's spec sheet. You can get a little insight to your disk drive with a utility that queries your drive using the SMART interface. I downloaded, sort of at random, GSmartControl to my Windows system. I think that this is a user interface for some command line utilities. Using GSmartControl, I can learn a lot about my disk drive. Of relevance, my hard drive apparently has 4096 byte physical sectors, but presents 512 byte logical sectors to the host. I think that hard drives have also had different physical and logical geometries (organization of the drive into cylinders and tracks) for quite some time. It seems to me that these physical/logical differences would conflict the classic disk operations optimizations that you should have learned about in your operating systems class, but... I'll let someone with an SSD run GSmartControl on their drive and verify that it, too, has 512 byte sectors. A disk management program from your hard drive vendor might tell you a lot more things about your hard drive. Hope this helps, -tjs From nompelis at nobelware.com Mon Oct 28 15:06:36 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Mon, 28 Oct 2019 10:06:36 -0500 Subject: Huh, I can't beat the Haskell performance in C In-Reply-To: <7d529522-0ce1-a746-c989-88c2e4b01401@saloits.com> References: <1781bf0e-1eb6-4788-889e-9cf50d73b919@www.fastmail.com> <2FF19050-D592-45CC-8ED4-E080DC654BF8@netjack.com> <20191023042040.GA40065@begriffs.com> <7d529522-0ce1-a746-c989-88c2e4b01401@saloits.com> Message-ID: <18913A7E-8A46-4DDF-9C0D-EC5F2663DF4F@nobelware.com> I trust magnetic media much MUCH more than SSD type drives.I like my Apple laptop be ause I throw it around but it is a mere terminal and browser in the end. Our admin has said that the most robust storage is still tape. Lots of things the consumer takes for granted with new storage technologies like SSD. Good info Tim. From drewbenson at netjack.com Mon Oct 28 15:15:50 2019 From: drewbenson at netjack.com (Andrew Benson) Date: Mon, 28 Oct 2019 10:15:50 -0500 Subject: Huh, I can't beat the Haskell performance in C In-Reply-To: <18913A7E-8A46-4DDF-9C0D-EC5F2663DF4F@nobelware.com> References: <1781bf0e-1eb6-4788-889e-9cf50d73b919@www.fastmail.com> <2FF19050-D592-45CC-8ED4-E080DC654BF8@netjack.com> <20191023042040.GA40065@begriffs.com> <7d529522-0ce1-a746-c989-88c2e4b01401@saloits.com> <18913A7E-8A46-4DDF-9C0D-EC5F2663DF4F@nobelware.com> Message-ID: <2F4F741A-8829-4560-BFAD-A451CE20D2E9@netjack.com> I trust backups. ?????? > On Oct 28, 2019, at 10:06 AM, Ioannis Nompelis wrote: > > I trust magnetic media much MUCH more than SSD type drives.I like my Apple laptop be ause I throw it around but it is a mere terminal and browser in the end. > > Our admin has said that the most robust storage is still tape. > > Lots of things the consumer takes for granted with new storage technologies like SSD. > > Good info Tim. From nompelis at nobelware.com Mon Oct 28 23:26:14 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Mon, 28 Oct 2019 23:26:14 +0000 Subject: Huh, I can't beat the Haskell performance in C In-Reply-To: References: <1781bf0e-1eb6-4788-889e-9cf50d73b919@www.fastmail.com> <2FF19050-D592-45CC-8ED4-E080DC654BF8@netjack.com> <20191023042040.GA40065@begriffs.com> <7d529522-0ce1-a746-c989-88c2e4b01401@saloits.com> <18913A7E-8A46-4DDF-9C0D-EC5F2663DF4F@nobelware.com> Message-ID: <20191028232614.GA5579@nobelware.com> Ha! Chris nailed it. But even spinning drives have caching issues and timeouts. The same admin told me that the manufacturers sell the same hard ware with different firmware at much higher prices, which are the "server rated" drives. The difference includes not faking writes and trully returning when data is written, and short timeouts. I am not sure of the details. I trust backups, which are as good as (a) your backup strategy, and most importantly the recovery strategy, and (b) your media on which it is made. The same admin plays a lot of tricks with exporting different directories from a given server so that they can appear as different drives to the server that does the backup. In this way, no backup is of the entire, say, 100TB drive, but rather the bacup of 20 "separate" disks at 5 TB each. I trust this man with my life. From joe at begriffs.com Tue Oct 29 02:18:10 2019 From: joe at begriffs.com (Joe Nelson) Date: Mon, 28 Oct 2019 21:18:10 -0500 Subject: HTML emails Message-ID: <20191029021810.rdjg7usaiobe3rs6@begriffs.com> Chris Pearson has been sending mail to this list, but it all got bounced with the message: The message's content type was not explicitly allowed It so happens that some of us have been able to see his messages because we're in the To or CC lines, but the list at large can't see those messages. Chris Mikkelson also told me on Thursday that it took him a while to configure his mailer to send plain text messages. So my question is, should I relax this restriction and allow HTML mail to be carried on the list? Mailman can also internally convert HTML to text, as well as "collapse" multipart/alternative to its first part content. I hesitate to have Mailman alter messages as that would fail DKIM checks for any of our domains that enforce them. Or do we continue to hold strong to our proud anachronism? From pstelzig at gmx.com Tue Oct 29 02:28:34 2019 From: pstelzig at gmx.com (pstelzig) Date: Mon, 28 Oct 2019 21:28:34 -0500 Subject: HTML emails In-Reply-To: <20191029021810.rdjg7usaiobe3rs6@begriffs.com> References: <20191029021810.rdjg7usaiobe3rs6@begriffs.com> Message-ID: <8fb78a7f-11e1-8ccd-d9c7-927bcaef584d@gmx.com> I still send plain text email in a corporate environment where the rule passed down from HR is everyone should use the corporate logo HTML signature. On 28/10/2019 21:18, Joe Nelson wrote: > Chris Pearson has been sending mail to this list, but it all got bounced > with the message: > > The message's content type was not explicitly allowed > > It so happens that some of us have been able to see his messages because > we're in the To or CC lines, but the list at large can't see those > messages. > > Chris Mikkelson also told me on Thursday that it took him a while to > configure his mailer to send plain text messages. > > So my question is, should I relax this restriction and allow HTML mail > to be carried on the list? Mailman can also internally convert HTML to > text, as well as "collapse" multipart/alternative to its first part > content. I hesitate to have Mailman alter messages as that would fail > DKIM checks for any of our domains that enforce them. > > Or do we continue to hold strong to our proud anachronism? > From zach at nor.mn Tue Oct 29 02:32:52 2019 From: zach at nor.mn (Zach Norman) Date: Mon, 28 Oct 2019 21:32:52 -0500 Subject: HTML emails In-Reply-To: <8fb78a7f-11e1-8ccd-d9c7-927bcaef584d@gmx.com> Message-ID: I convert html mail back down to plain text in my client. It seems overly cumbersome for something we can use social pressure and other tools to encourage. Zach normnco automata ? Original Message ? From: pstelzig at gmx.com Sent: October 28, 2019 21:28 To: friends at talk.begriffs.com Subject: Re: HTML emails I still send plain text email in a corporate environment where the rule passed down from HR is everyone should use the corporate logo HTML signature. On 28/10/2019 21:18, Joe Nelson wrote: > Chris Pearson has been sending mail to this list, but it all got bounced > with the message: > > The message's content type was not explicitly allowed > > It so happens that some of us have been able to see his messages because > we're in the To or CC lines, but the list at large can't see those > messages. > > Chris Mikkelson also told me on Thursday that it took him a while to > configure his mailer to send plain text messages. > > So my question is, should I relax this restriction and allow HTML mail > to be carried on the list? Mailman can also internally convert HTML to > text, as well as "collapse" multipart/alternative to its first part > content. I hesitate to have Mailman alter messages as that would fail > DKIM checks for any of our domains that enforce them. > > Or do we continue to hold strong to our proud anachronism? > From denisovenator at gmail.com Tue Oct 29 02:33:30 2019 From: denisovenator at gmail.com (Victor Denisov) Date: Mon, 28 Oct 2019 19:33:30 -0700 Subject: HTML emails In-Reply-To: <20191029021810.rdjg7usaiobe3rs6@begriffs.com> References: <20191029021810.rdjg7usaiobe3rs6@begriffs.com> Message-ID: I suggest to hold strong to our proud anachronism. There is something beautiful in reading plain text emails. On Mon, Oct 28, 2019 at 7:18 PM Joe Nelson wrote: > > Chris Pearson has been sending mail to this list, but it all got bounced > with the message: > > The message's content type was not explicitly allowed > > It so happens that some of us have been able to see his messages because > we're in the To or CC lines, but the list at large can't see those > messages. > > Chris Mikkelson also told me on Thursday that it took him a while to > configure his mailer to send plain text messages. > > So my question is, should I relax this restriction and allow HTML mail > to be carried on the list? Mailman can also internally convert HTML to > text, as well as "collapse" multipart/alternative to its first part > content. I hesitate to have Mailman alter messages as that would fail > DKIM checks for any of our domains that enforce them. > > Or do we continue to hold strong to our proud anachronism? From louis at goessling.com Tue Oct 29 03:26:01 2019 From: louis at goessling.com (Louis) Date: Mon, 28 Oct 2019 22:26:01 -0500 Subject: HTML emails In-Reply-To: References: <20191029021810.rdjg7usaiobe3rs6@begriffs.com> Message-ID: My vote (what it weighs due to my infrequent posting is up for debate though... haha) is for proud anachronism and continued use of plaintext mail in all it's austere beauty. Louis Goessling On Mon, Oct 28, 2019 at 9:33 PM Victor Denisov wrote: > > I suggest to hold strong to our proud anachronism. > There is something beautiful in reading plain text emails. > > > On Mon, Oct 28, 2019 at 7:18 PM Joe Nelson wrote: > > > > Chris Pearson has been sending mail to this list, but it all got bounced > > with the message: > > > > The message's content type was not explicitly allowed > > > > It so happens that some of us have been able to see his messages because > > we're in the To or CC lines, but the list at large can't see those > > messages. > > > > Chris Mikkelson also told me on Thursday that it took him a while to > > configure his mailer to send plain text messages. > > > > So my question is, should I relax this restriction and allow HTML mail > > to be carried on the list? Mailman can also internally convert HTML to > > text, as well as "collapse" multipart/alternative to its first part > > content. I hesitate to have Mailman alter messages as that would fail > > DKIM checks for any of our domains that enforce them. > > > > Or do we continue to hold strong to our proud anachronism? From kermit4 at gmail.com Tue Oct 29 04:44:45 2019 From: kermit4 at gmail.com (Christopher Pearson) Date: Mon, 28 Oct 2019 23:44:45 -0500 Subject: Huh, I can't beat the Haskell performance in C In-Reply-To: References: <1781bf0e-1eb6-4788-889e-9cf50d73b919@www.fastmail.com> <2FF19050-D592-45CC-8ED4-E080DC654BF8@netjack.com> <20191023042040.GA40065@begriffs.com> <7d529522-0ce1-a746-c989-88c2e4b01401@saloits.com> <18913A7E-8A46-4DDF-9C0D-EC5F2663DF4F@nobelware.com> <20191028232614.GA5579@nobelware.com> Message-ID: On Mon, Oct 28, 2019 at 6:26 PM Ioannis Nompelis wrote: > > Ha! Chris nailed it. > > The same admin told > me that the manufacturers sell the same hard ware with different firmware at > much higher prices, which are the "server rated" drives. The difference > includes not faking writes and trully returning when data is written, and > short timeouts. I don't think any spindles fake writes. The timeout/enterprise thing is that drives for RAID support short timeouts so you can get the sector from redundancy rather than time out and boot the drive My original reply which didn't make it to the list due to content-type: >> >> I trust magnetic media much MUCH more than SSD type drives. > > Yeah, most cheat benchmarks by faking confirmation of being written to permanent storage, leaving data in their cache, losing data in power loss or even a crash of its own OS. Also often bombing on sustained writes and mixed I/O, and you don't know you're near the throughput limit until you hit it as you're only seeing the time to write to the cache. Spindles are honest, and get more efficient under load (physical reordering) so you don't hit a throughput wall suddenly. If you want to write to RAM just buy more RAM and don't call sync as much. From kermit4 at gmail.com Tue Oct 29 04:46:12 2019 From: kermit4 at gmail.com (Christopher Pearson) Date: Mon, 28 Oct 2019 23:46:12 -0500 Subject: HTML emails In-Reply-To: References: <20191029021810.rdjg7usaiobe3rs6@begriffs.com> Message-ID: plain/text is fine with me, I was just confused at first because some were getting my messages (they were cc'd). I just have to click 3 dots and select plain text mode in gmail before sending. I thought there was a default somewhere for plain/text which I thought I had on, but can't find it now. From nompelis at nobelware.com Tue Oct 29 12:36:27 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Tue, 29 Oct 2019 07:36:27 -0500 Subject: HTML emails In-Reply-To: References: <20191029021810.rdjg7usaiobe3rs6@begriffs.com> Message-ID: I like plain text emails. The main problem is the DKIM and I do not have a solution to offer other than making it a clause on the sign up pave. From nompelis at nobelware.com Tue Oct 29 12:38:19 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Tue, 29 Oct 2019 07:38:19 -0500 Subject: HTML emails In-Reply-To: References: <20191029021810.rdjg7usaiobe3rs6@begriffs.com> Message-ID: <35AE2790-4D77-4FD5-BDD7-11FBAA3A5704@nobelware.com> There may be a hotkey for switching to plain on gmail. From dave.bucklin at gmail.com Tue Oct 29 12:47:17 2019 From: dave.bucklin at gmail.com (Dave Bucklin) Date: Tue, 29 Oct 2019 07:47:17 -0500 Subject: HTML emails In-Reply-To: References: <20191029021810.rdjg7usaiobe3rs6@begriffs.com> Message-ID: <20191029124717.GA17921@19a6.tech> On Tue, Oct 29, 2019 at 07:36:27AM -0500, Ioannis Nompelis wrote: > I like plain text emails. > > The main problem is the DKIM and I do not have a solution to offer other than making it a clause on the sign up pave. Many (all?) of my GNU mailing lists have switched to non-munging in deference to DKIM. From hello at robertdherb.com Tue Oct 29 13:45:06 2019 From: hello at robertdherb.com (Robbie H) Date: Tue, 29 Oct 2019 08:45:06 -0500 Subject: HTML emails In-Reply-To: References: <8fb78a7f-11e1-8ccd-d9c7-927bcaef584d@gmx.com> Message-ID: <20191029084506.6fe03932@phoebe.vault-423.qualityretro.net> On Mon, 28 Oct 2019 21:32:52 -0500 Zach Norman wrote: > It seems > overly cumbersome for something we can use social pressure and other > tools to encourage. > > Zach > > normnco automata > On the other hand, social pressure doesn't seem to stop top-posting. =) From pstelzig at gmx.com Tue Oct 29 14:00:12 2019 From: pstelzig at gmx.com (pstelzig) Date: Tue, 29 Oct 2019 09:00:12 -0500 Subject: HTML emails In-Reply-To: <20191029084506.6fe03932@phoebe.vault-423.qualityretro.net> References: <8fb78a7f-11e1-8ccd-d9c7-927bcaef584d@gmx.com> <20191029084506.6fe03932@phoebe.vault-423.qualityretro.net> Message-ID: On 29/10/2019 08:45, Robbie H wrote: > On Mon, 28 Oct 2019 21:32:52 -0500 > Zach Norman wrote: > >> It seems >> overly cumbersome for something we can use social pressure and other >> tools to encourage. >> >> Zach >> >> normnco automata >> > > On the other hand, social pressure doesn't seem to stop top-posting. =) > Sorry about that. From nompelis at nobelware.com Tue Oct 29 14:13:59 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Tue, 29 Oct 2019 14:13:59 +0000 Subject: Huh, I can't beat the Haskell performance in C In-Reply-To: References: <2FF19050-D592-45CC-8ED4-E080DC654BF8@netjack.com> <20191023042040.GA40065@begriffs.com> <7d529522-0ce1-a746-c989-88c2e4b01401@saloits.com> <18913A7E-8A46-4DDF-9C0D-EC5F2663DF4F@nobelware.com> <20191028232614.GA5579@nobelware.com> Message-ID: <20191029141359.GA7533@nobelware.com> > > I don't think any spindles fake writes. The timeout/enterprise thing > is that drives for RAID support short timeouts so you can get the > sector from redundancy rather than time out and boot the drive > That's right. RAID is the magic word, when multiple drives have to play nice together. By the way, if you have suggestions for what drives (enterprise level) I should be getting to update my old ones, anyone, please let me know. From mxu at uribe.cc Tue Oct 29 15:02:58 2019 From: mxu at uribe.cc (Mauricio Uribe) Date: Tue, 29 Oct 2019 11:02:58 -0400 Subject: HTML emails In-Reply-To: <20191029021810.rdjg7usaiobe3rs6@begriffs.com> References: <20191029021810.rdjg7usaiobe3rs6@begriffs.com> Message-ID: > > Chris Pearson has been sending mail to this list, but it all got bounced > with the message: > > The message's content type was not explicitly allowed > > It so happens that some of us have been able to see his messages because > we're in the To or CC lines, but the list at large can't see those > messages. > > Chris Mikkelson also told me on Thursday that it took him a while to > configure his mailer to send plain text messages. > > So my question is, should I relax this restriction and allow HTML mail > to be carried on the list? Mailman can also internally convert HTML to > text, as well as "collapse" multipart/alternative to its first part > content. I hesitate to have Mailman alter messages as that would fail > DKIM checks for any of our domains that enforce them. > > Or do we continue to hold strong to our proud anachronism? I vote plain text...but html is fine too if that is what the group prefers. (I know, what a pushover i am, right?) As it is gmail client makes it annoying anyway. From drewbenson at netjack.com Tue Oct 29 15:04:17 2019 From: drewbenson at netjack.com (Andrew Benson) Date: Tue, 29 Oct 2019 10:04:17 -0500 Subject: HTML emails In-Reply-To: References: <20191029021810.rdjg7usaiobe3rs6@begriffs.com> Message-ID: <1598C71C-DAF5-494B-A1F9-728D6AA509D2@netjack.com> Relax the restriction. > On Oct 29, 2019, at 10:02 AM, Mauricio Uribe wrote: > >> >> Chris Pearson has been sending mail to this list, but it all got bounced >> with the message: >> >> The message's content type was not explicitly allowed >> >> It so happens that some of us have been able to see his messages because >> we're in the To or CC lines, but the list at large can't see those >> messages. >> >> Chris Mikkelson also told me on Thursday that it took him a while to >> configure his mailer to send plain text messages. >> >> So my question is, should I relax this restriction and allow HTML mail >> to be carried on the list? Mailman can also internally convert HTML to >> text, as well as "collapse" multipart/alternative to its first part >> content. I hesitate to have Mailman alter messages as that would fail >> DKIM checks for any of our domains that enforce them. >> >> Or do we continue to hold strong to our proud anachronism? > > I vote plain text...but html is fine too if that is what the group > prefers. (I know, what a pushover i am, right?) As it is gmail > client makes it annoying anyway. From joe at begriffs.com Tue Oct 29 15:37:57 2019 From: joe at begriffs.com (Joe Nelson) Date: Tue, 29 Oct 2019 10:37:57 -0500 Subject: HTML emails In-Reply-To: <1598C71C-DAF5-494B-A1F9-728D6AA509D2@netjack.com> References: <20191029021810.rdjg7usaiobe3rs6@begriffs.com> <1598C71C-DAF5-494B-A1F9-728D6AA509D2@netjack.com> Message-ID: <20191029153757.GC59064@begriffs.com> Andrew Benson wrote: > Relax the restriction. It appears that most people on the list prefer plain text, so perhaps the question is: how hard is it for different mail clients to send in plain text mode? Looks like you're using Apple Mail via mailanyone.net -- was it hard for you to configure it to send as plain text? Here are the steps to do it with various mail clients: * K-9 for Android: change "Message Format" under account settings https://k9mail.github.io/documentation/settings/account.html * Gmail web client: plain text mode is behind a "..." icon https://www.lifewire.com/how-to-send-a-message-in-plain-text-from-gmail-1171963 * Mac Mail: from the Format menu https://support.apple.com/guide/mail/use-plain-or-rich-text-in-emails-mlhlp1009/mac From drewbenson at netjack.com Tue Oct 29 16:40:13 2019 From: drewbenson at netjack.com (Andrew Benson) Date: Tue, 29 Oct 2019 11:40:13 -0500 Subject: HTML emails In-Reply-To: <20191029153757.GC59064@begriffs.com> References: <20191029153757.GC59064@begriffs.com> Message-ID: Actually I think it figured it out on its own in some way. When I was emailing from my Mac previously, it was getting rejected every time. Now it?s not. Never took the time to figure it out lol. Interestingly, it always worked from my phone. > > On Oct 29, 2019, at 10:38 AM, Joe Nelson wrote: > ?Andrew Benson wrote: > Relax the restriction. It appears that most people on the list prefer plain text, so perhaps the question is: how hard is it for different mail clients to send in plain text mode? Looks like you're using Apple Mail via mailanyone.net -- was it hard for you to configure it to send as plain text? Here are the steps to do it with various mail clients: * K-9 for Android: change "Message Format" under account settings https://k9mail.github.io/documentation/settings/account.html * Gmail web client: plain text mode is behind a "..." icon https://www.lifewire.com/how-to-send-a-message-in-plain-text-from-gmail-1171963 * Mac Mail: from the Format menu https://support.apple.com/guide/mail/use-plain-or-rich-text-in-emails-mlhlp1009/mac From drewbenson at netjack.com Tue Oct 29 16:43:54 2019 From: drewbenson at netjack.com (Andrew Benson) Date: Tue, 29 Oct 2019 11:43:54 -0500 Subject: HTML emails In-Reply-To: <20191029153757.GC59064@begriffs.com> References: <20191029153757.GC59064@begriffs.com> Message-ID: <3E85A92F-4952-4BF4-8EA7-7CB445B3B4BD@netjack.com> Often the issue with non-plaintext emails is that some people make ridiculous decisions, like a multi-colored plaid background with yellow text, for example, ruining everyone?s lives. I had written ?relax the description? because I don?t expect (and haven?t seen) anyone here to do anything so foolish. > > On Oct 29, 2019, at 10:38 AM, Joe Nelson wrote: > ?Andrew Benson wrote: > Relax the restriction. It appears that most people on the list prefer plain text, so perhaps the question is: how hard is it for different mail clients to send in plain text mode? Looks like you're using Apple Mail via mailanyone.net -- was it hard for you to configure it to send as plain text? Here are the steps to do it with various mail clients: * K-9 for Android: change "Message Format" under account settings https://k9mail.github.io/documentation/settings/account.html * Gmail web client: plain text mode is behind a "..." icon https://www.lifewire.com/how-to-send-a-message-in-plain-text-from-gmail-1171963 * Mac Mail: from the Format menu https://support.apple.com/guide/mail/use-plain-or-rich-text-in-emails-mlhlp1009/mac From salo at saloits.com Tue Oct 29 18:03:03 2019 From: salo at saloits.com (Timothy J. Salo) Date: Tue, 29 Oct 2019 13:03:03 -0500 Subject: HTML emails In-Reply-To: <20191029153757.GC59064@begriffs.com> References: <20191029021810.rdjg7usaiobe3rs6@begriffs.com> <1598C71C-DAF5-494B-A1F9-728D6AA509D2@netjack.com> <20191029153757.GC59064@begriffs.com> Message-ID: <349397b1-6227-2885-ef91-de673f1b6f7f@saloits.com> On 10/29/2019 10:37 AM, Joe Nelson wrote: > Andrew Benson wrote: >> Relax the restriction. > > It appears that most people on the list prefer plain text, so perhaps > the question is: how hard is it for different mail clients to send in > plain text mode? It seems to me to be a matter of objectives. If the objective is to facilitate communications among interested parties, then I believe that the right approach is to accept pretty much anything that looks even remotely like email. Filtering email based on syntax, (e.g., text versus HTML), doesn't seem supportive of effective group communications. But, perhaps I don't really understand the objective of this group. -tjs From nompelis at nobelware.com Tue Oct 29 19:22:48 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Tue, 29 Oct 2019 19:22:48 +0000 Subject: HTML emails In-Reply-To: <3E85A92F-4952-4BF4-8EA7-7CB445B3B4BD@netjack.com> References: <20191029153757.GC59064@begriffs.com> <3E85A92F-4952-4BF4-8EA7-7CB445B3B4BD@netjack.com> Message-ID: <20191029192248.GA19212@nobelware.com> > > I had written ?relax the description? because I don?t expect (and haven?t seen) anyone here to do anything so foolish. > Like Christina Ricci said as Wednesday Addams: "Wait..." From chris at mikk.net Tue Oct 29 19:46:52 2019 From: chris at mikk.net (Chris Mikkelson) Date: Tue, 29 Oct 2019 14:46:52 -0500 Subject: HTML emails In-Reply-To: <20191029021810.rdjg7usaiobe3rs6@begriffs.com> References: <20191029021810.rdjg7usaiobe3rs6@begriffs.com> Message-ID: <20191029194652.GA51347@mikk.net> On Mon, Oct 28, 2019 at 09:18:10PM -0500, Joe Nelson wrote: > Chris Mikkelson also told me on Thursday that it took him a while to > configure his mailer to send plain text messages. FWIW, "took me a while to" was more "took me a while to get around to it", since as a CLI lover, I find hunting through menus to be less pleasant than reading through man pages. > So my question is, should I relax this restriction and allow HTML mail > to be carried on the list? Mailman can also internally convert HTML to > text, as well as "collapse" multipart/alternative to its first part > content. I hesitate to have Mailman alter messages as that would fail > DKIM checks for any of our domains that enforce them. > > Or do we continue to hold strong to our proud anachronism? I'm on the "hold strong" side. Text/plain messages can be read anywhere by any mail reader north of: cat /var/spool/mail/$(whoami) in complexity. Reading text/html requires certain mail readers, or hacks. I use a hack[1] which work well, but only cases where the html message could have been a text/plain one in the first place. -- Chris Mikkelson | That is a form of errant pedantry up with which chris at mikk.net | I will not put. | -- Winston Churchill [1] In particular, splicing 'lynx -dump' into the display path of whatever mailer I'm using. Mutt's explicit MIME support lets you do this with ~/.mailcap, but mh (which I used earlier, and miss) required slightly more involved integration. From joe at begriffs.com Wed Oct 30 01:11:16 2019 From: joe at begriffs.com (Joe Nelson) Date: Tue, 29 Oct 2019 20:11:16 -0500 Subject: HTML emails In-Reply-To: <349397b1-6227-2885-ef91-de673f1b6f7f@saloits.com> References: <20191029021810.rdjg7usaiobe3rs6@begriffs.com> <1598C71C-DAF5-494B-A1F9-728D6AA509D2@netjack.com> <20191029153757.GC59064@begriffs.com> <349397b1-6227-2885-ef91-de673f1b6f7f@saloits.com> Message-ID: <20191030011116.GD59064@begriffs.com> > If the objective is to facilitate communications among interested > parties, then I believe that the right approach is to accept pretty > much anything that looks even remotely like email. Fair point. I'm interested not just in the volume of messages, but in their tone. Theoretically any place people can type messages should support the same conversations, but different media seems to promote different styles. For instance Slack rooms often get filled up with party parrot emojis and animated gifs (see for example, the local Slack rooms "For the Horde" and "MSPTech"). Removing the ability to add styles, colors, and pictures makes discussions more serious. And email itself seems to encourage long-form writing. At least longer than chat programs. Adding a rule to prevent "rich text" can also help people stop and learn more about email, rather than thinking of it as a word processor with a send button. They can learn more about MIME. (And I can learn too, for that matter, such as the adjustment I just made to try to allow you to attach C source files!) I asked earlier about enabling HTML mail because it seemed that two people I respect found it difficult to use. However it sounds like they just hadn't gotten around to configuring their mailers. That said, mailman's error message about the content type not being explicitly allowed is pretty cryptic. Maybe there's a way to update the code to add a hint to the message. From joe at begriffs.com Wed Oct 30 05:02:49 2019 From: joe at begriffs.com (Joe Nelson) Date: Wed, 30 Oct 2019 00:02:49 -0500 Subject: Testing whether I can attach source code Message-ID: <20191030050249.GE59064@begriffs.com> If you can see this message, then I got the content type whitelist right. -------------- next part -------------- int main(void) { return 0; } From joe at begriffs.com Wed Oct 30 05:20:45 2019 From: joe at begriffs.com (Joe Nelson) Date: Wed, 30 Oct 2019 00:20:45 -0500 Subject: Testing whether I can attach source code In-Reply-To: <20191030050249.GE59064@begriffs.com> References: <20191030050249.GE59064@begriffs.com> Message-ID: <20191030052045.GF59064@begriffs.com> > If you can see this message, then I got the content type whitelist right. > int main(void) > { > return 0; > } Nice, that worked. Also, apple mail is a piece of crap. Because I marked "Content-Disposition: inline" for the message and for the attachment, apple mail displays them together in the message body (which is good) but shows no indication that there is an attachment with suggested name "fun.c" which can be saved. Mutt shows this very clearly. I'm curious what other people on the list see in their mail clients. From pstelzig at gmx.com Wed Oct 30 13:06:40 2019 From: pstelzig at gmx.com (pstelzig) Date: Wed, 30 Oct 2019 08:06:40 -0500 Subject: Testing whether I can attach source code In-Reply-To: <20191030052045.GF59064@begriffs.com> References: <20191030050249.GE59064@begriffs.com> <20191030052045.GF59064@begriffs.com> Message-ID: <0b7e686c-94d5-1c50-6f79-3318acee2456@gmx.com> On 30/10/2019 00:20, Joe Nelson wrote: >> If you can see this message, then I got the content type whitelist right. > >> int main(void) >> { >> return 0; >> } > > Nice, that worked. > > Also, apple mail is a piece of crap. Because I marked > "Content-Disposition: inline" for the message and for the attachment, > apple mail displays them together in the message body (which is good) > but shows no indication that there is an attachment with suggested name > "fun.c" which can be saved. Mutt shows this very clearly. > > I'm curious what other people on the list see in their mail clients. > Thunderbird showed it inline and had an attachment of fun.c From hello at robertdherb.com Wed Oct 30 13:37:05 2019 From: hello at robertdherb.com (Robbie H) Date: Wed, 30 Oct 2019 08:37:05 -0500 Subject: Testing whether I can attach source code In-Reply-To: <20191030052045.GF59064@begriffs.com> References: <20191030050249.GE59064@begriffs.com> <20191030052045.GF59064@begriffs.com> Message-ID: <20191030083705.064486e1@phoebe.vault-423.qualityretro.net> On Wed, 30 Oct 2019 00:20:45 -0500 Joe Nelson wrote: > I'm curious what other people on the list see in their mail clients. K-9 showed it as an attachment, but Claws showed it inline. If I were paying attention, I would have noticed there was an attachment I could download, but Claws' UI doesn't exactly prioritize attachments, so I would have very easily missed it. I find it interesting that mutt (And maybe other clients would too?) converted the attachment to text when you replied. From andrew at apastron.io Wed Oct 30 13:39:44 2019 From: andrew at apastron.io (Andrew Fischer) Date: Wed, 30 Oct 2019 08:39:44 -0500 Subject: Testing whether I can attach source code In-Reply-To: <20191030052045.GF59064@begriffs.com> Message-ID: On Wed Oct 30, 2019 at 12:20 AM Joe Nelson wrote: > I'm curious what other people on the list see in their mail clients. The aerc mail client showed the message while the c file appears as another plain/text attachment. From joe at begriffs.com Wed Oct 30 14:30:06 2019 From: joe at begriffs.com (Joe Nelson) Date: Wed, 30 Oct 2019 09:30:06 -0500 Subject: Testing whether I can attach source code In-Reply-To: <20191030083705.064486e1@phoebe.vault-423.qualityretro.net> References: <20191030050249.GE59064@begriffs.com> <20191030052045.GF59064@begriffs.com> <20191030083705.064486e1@phoebe.vault-423.qualityretro.net> Message-ID: <20191030143006.GH59064@begriffs.com> Robbie H wrote: > I find it interesting that mutt (And maybe other clients would too?) > converted the attachment to text when you replied. I think that's a feature for code review. If the sender marks the source code attachment as inline then recipients can more easily reply and talk about quoted lines/sections. That's how the PostgreSQL community works, they attach patch files (preferably inline if the patch isn't too huge), and then people reply about various sections. By contrast, the Linux, OpenBSD and FreeBSD development mailing lists have the policy, ?No MIME, no links, no compression, no attachments. Just plain text.? The patch goes right in the body of the message at the bottom. If the mail client would mangle the code (such as by wrapping long lines), then the lists permit attaching with MIME. From mxu at uribe.cc Wed Oct 30 14:39:43 2019 From: mxu at uribe.cc (Mauricio Uribe) Date: Wed, 30 Oct 2019 10:39:43 -0400 Subject: Testing whether I can attach source code In-Reply-To: References: <20191030052045.GF59064@begriffs.com> Message-ID: On Wed, Oct 30, 2019 at 9:39 AM Andrew Fischer wrote: > > On Wed Oct 30, 2019 at 12:20 AM Joe Nelson wrote: > > I'm curious what other people on the list see in their mail clients. > > The aerc mail client showed the message while the c file > appears as another plain/text attachment. Gmail (actually "Google For your Domain", or whatever it is called nowadays) seemed to behave like aerc, as Andrew has just noted. Side note: When I'm at home, I tend to use thunderbird connected to this account, but have been interested in aerc. Hmmm. - mauricio From nompelis at nobelware.com Wed Oct 30 14:41:45 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Wed, 30 Oct 2019 14:41:45 +0000 Subject: HTML emails In-Reply-To: <20191030011116.GD59064@begriffs.com> References: <20191029021810.rdjg7usaiobe3rs6@begriffs.com> <1598C71C-DAF5-494B-A1F9-728D6AA509D2@netjack.com> <20191029153757.GC59064@begriffs.com> <349397b1-6227-2885-ef91-de673f1b6f7f@saloits.com> <20191030011116.GD59064@begriffs.com> Message-ID: <20191030144145.GA5125@nobelware.com> Discussions being serious should not be confused with discussions that contain no humour! From nompelis at nobelware.com Wed Oct 30 14:47:07 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Wed, 30 Oct 2019 14:47:07 +0000 Subject: Testing whether I can attach source code In-Reply-To: <20191030143006.GH59064@begriffs.com> References: <20191030050249.GE59064@begriffs.com> <20191030052045.GF59064@begriffs.com> <20191030083705.064486e1@phoebe.vault-423.qualityretro.net> <20191030143006.GH59064@begriffs.com> Message-ID: <20191030144707.GB5125@nobelware.com> Mutt here as well, marking inlined and inlining the text. Mutt can also inline HTML (but not the version I am using on this system). I would not mimic the large development lists policies. I really like what Joe has done with this list so far. And by the way, I get the sense that it has been growing and that a lot of expertise is connected here. (W00T!) We may have to go through another round of introductions, perhaps on our website, which we have yet to actually put together. From drewbenson at netjack.com Wed Oct 30 14:48:24 2019 From: drewbenson at netjack.com (Andrew Benson) Date: Wed, 30 Oct 2019 09:48:24 -0500 Subject: Testing whether I can attach source code In-Reply-To: <20191030050249.GE59064@begriffs.com> References: <20191030050249.GE59064@begriffs.com> Message-ID: <66B13623-EA5E-40DA-967E-C921A16C7D2E@netjack.com> Mac mail (and iOS Mail) both show text in-line, but show no attachments. > On Oct 30, 2019, at 12:02 AM, Joe Nelson wrote: > > If you can see this message, then I got the content type whitelist right. > int main(void) > { > return 0; > } From joe at begriffs.com Wed Oct 30 15:20:08 2019 From: joe at begriffs.com (Joe Nelson) Date: Wed, 30 Oct 2019 10:20:08 -0500 Subject: Testing whether I can attach source code In-Reply-To: References: <20191030052045.GF59064@begriffs.com> Message-ID: <20191030152008.GI59064@begriffs.com> Mauricio Uribe wrote: > have been interested in aerc. Hmmm. For those who have used aerc, I'm curious if you feel it lives up to its self-described "world's best email client" moniker. I personally don't see what it offers over mutt. The aerc web page says: > Editing emails in an embedded terminal tmux-style, allowing you to check on > incoming emails and reference other threads while you compose your replies So open two instances of mutt. It's lightweight and they can both access the same maildir on disk at the same time. > Render HTML emails with an interactive terminal web browser, highlight > patches with diffs, and browse with an embedded less session Yeah, that's what .mailcap is for, to associate content types with viewers. This is standard mutt. > Vim-style keybindings and ex-command system, allowing for powerful automation > at a single keystroke See "bind" in .muttrc > First-class support for working with git & email In that you can pipe messages to a helper program? Again, standard. > Open a new tab with a terminal emulator and a shell running for easy access > to nearby git repos for parallel work This is what a window manager or terminal multiplexer are for. Why would he duplicate this functionality in a mail reader? > Support for multiple accounts, with support for IMAP, Maildir, SMTP, and > sendmail transfer protocols I use a program called mbsync to reconcile a local maildir with an imap server. Mutt deals with the maildir and I get offline access to all my emails. This supports multiple accounts. > Asynchronous IMAP support ensures the UI never gets locked up by a flaky > network, as mutt often does Interesting, I haven't tried pulling the mail directly from mutt so this may be the first actual problem he identified. Not a problem in my setup though. > Efficient network usage - aerc only downloads the information which is > necessary to present the UI, making for a snappy and bandwidth-efficient > experience I'm curious how fast it goes for a truly high-volume list. Like I said, I pull from a local maildir, and have the mutt header_cache enabled. > 100% free and open source software! Speaking of open source, it kind of bothers me that this guy didn't take the time to learn about existing tools. He could have spent his energy contributing to mutt. For instance, to improve the imap performance problems he talked about. Having the patience to interact with their development community would probably have been more instructive. , I'll get off my soapbox, people can work on whatever they want. From drewbenson at netjack.com Wed Oct 30 15:20:41 2019 From: drewbenson at netjack.com (Andrew Benson) Date: Wed, 30 Oct 2019 10:20:41 -0500 Subject: HTML emails In-Reply-To: <20191030011116.GD59064@begriffs.com> References: <20191029021810.rdjg7usaiobe3rs6@begriffs.com> <1598C71C-DAF5-494B-A1F9-728D6AA509D2@netjack.com> <20191029153757.GC59064@begriffs.com> <349397b1-6227-2885-ef91-de673f1b6f7f@saloits.com> <20191030011116.GD59064@begriffs.com> Message-ID: <1C5AD9B8-8E55-4AFA-A4F3-79D53FB37CF9@netjack.com> The error message I was getting back didn?t seem to indicate that I needed to change anything on my end ? it really just looked like your mailer was broken. I expect that?s how most (new) people would take it. I understand what you mean with Slack getting filled with garbage. I think this is generally a difference in how email versus instant messaging is used, and I think Slack ? any slack room ? is worse than most in this way, for whatever reason. That said, people don?t tend to use email that way, and people communicating in a technically-specific mailing list are even less likely to do so. While I think everyone will agree that we don?t need to see flashing GIFs and plaid wallpaper behind orange text from some crazy person, plaintext can really be enhanced with some basic formatting features. Honestly, I think Markdown gets it almost exactly right. You can do bold, italics, fixed-width code formatting, but no Calligraphy fonts with yellow text on white background. Unfortunately that?s not really an option. Richtext leaves room for abuse, but certainly less so than HTML. Question: Have there been problems with people doing weird custom fonts/colors/etc in the past? Personally I find the ability to italicize a word or phrase, or maybe mark them with bold to be what I use most often to better convey where meaning an emphasis should be. For example, in the second paragraph above, I would like to have put the word ?tend? in italics. Thanks! > On Oct 29, 2019, at 8:11 PM, Joe Nelson wrote: > >> If the objective is to facilitate communications among interested >> parties, then I believe that the right approach is to accept pretty >> much anything that looks even remotely like email. > > Fair point. I'm interested not just in the volume of messages, but in > their tone. Theoretically any place people can type messages should > support the same conversations, but different media seems to promote > different styles. For instance Slack rooms often get filled up with > party parrot emojis and animated gifs (see for example, the local Slack > rooms "For the Horde" and "MSPTech"). > > Removing the ability to add styles, colors, and pictures makes > discussions more serious. And email itself seems to encourage long-form > writing. At least longer than chat programs. > > Adding a rule to prevent "rich text" can also help people stop and learn > more about email, rather than thinking of it as a word processor with a > send button. They can learn more about MIME. (And I can learn too, for > that matter, such as the adjustment I just made to try to allow you to > attach C source files!) > > I asked earlier about enabling HTML mail because it seemed that two > people I respect found it difficult to use. However it sounds like they > just hadn't gotten around to configuring their mailers. That said, > mailman's error message about the content type not being explicitly > allowed is pretty cryptic. Maybe there's a way to update the code to > add a hint to the message. From nompelis at nobelware.com Wed Oct 30 16:29:53 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Wed, 30 Oct 2019 16:29:53 +0000 Subject: Testing whether I can attach source code In-Reply-To: <20191030152008.GI59064@begriffs.com> References: <20191030052045.GF59064@begriffs.com> <20191030152008.GI59064@begriffs.com> Message-ID: <20191030162953.GA9300@nobelware.com> First of all, this: > I'll get off my soapbox, people can work on whatever they want. > And second, if it were not for people creating _entirely_ different codebases, we would all be living in Aldus-Huxley-ian dystopian computer world. We would also be driving only LADAs. From dave.bucklin at gmail.com Wed Oct 30 16:35:17 2019 From: dave.bucklin at gmail.com (Dave Bucklin) Date: Wed, 30 Oct 2019 11:35:17 -0500 Subject: Testing whether I can attach source code In-Reply-To: <20191030143006.GH59064@begriffs.com> References: <20191030050249.GE59064@begriffs.com> <20191030052045.GF59064@begriffs.com> <20191030083705.064486e1@phoebe.vault-423.qualityretro.net> <20191030143006.GH59064@begriffs.com> Message-ID: On October 30, 2019 9:30:06 AM CDT, Joe Nelson wrote: >Robbie H wrote: >> I find it interesting that mutt (And maybe other clients would too?) >> converted the attachment to text when you replied. That's what K-9 Mail did, too. From dave.bucklin at gmail.com Wed Oct 30 17:22:16 2019 From: dave.bucklin at gmail.com (Dave Bucklin) Date: Wed, 30 Oct 2019 12:22:16 -0500 Subject: Testing whether I can attach source code In-Reply-To: <20191030162953.GA9300@nobelware.com> References: <20191030052045.GF59064@begriffs.com> <20191030152008.GI59064@begriffs.com> <20191030162953.GA9300@nobelware.com> Message-ID: <5BD375C7-308A-4188-84E3-83DD3D500D61@gmail.com> On October 30, 2019 11:29:53 AM CDT, Ioannis Nompelis wrote: >First of all, this: > >> I'll get off my soapbox, people can work on whatever they want. >> > >And second, if it were not for people creating _entirely_ different >codebases, we would all be living in Aldus-Huxley-ian dystopian >computer >world. We would also be driving only LADAs. When asked if he was bothered by NeoVim, Bram said something like, no, if they do something useful, I can just copy it. So, it seems the mutt folks can decide whether these are problems they want to address and can look at the code as a starting point. Also, have you seen the new LADAs? ?? From andrew at apastron.io Wed Oct 30 17:27:58 2019 From: andrew at apastron.io (Andrew Fischer) Date: Wed, 30 Oct 2019 12:27:58 -0500 Subject: Testing whether I can attach source code In-Reply-To: <20191030152008.GI59064@begriffs.com> Message-ID: On Wed Oct 30, 2019 at 10:20 AM Joe Nelson wrote: > For those who have used aerc, I'm curious if you feel it lives up to its > self-described "world's best email client" moniker. I personally don't see what > it offers over mutt. I used mutt in the early 2000s. Then used thunderbird and web based gmail for some time. Finally last year got tired of the constant gmail interface changes and features I found unhelpful, plus I live almost purely in the shell anyway, so I went back to mutt. I actually don't find any of the features listed for aerc to be significantly different than mutt. I'm a bit embarassed to admit I only switched because I strugged with mutt's configuration (my 20-odd years of unix-y experience notwithstanding). When I went back to mutt I had multiple email accounts, all on my own mailserver: opensmdp + dovecot + rspamd + etc. And I was having a much harder time than I thought I should in setting up mutt to use multiple accounts together and, in particular, send mail from the appropriate account. I found aerc after a brief hunt for alternatives. The account configuration was dead simple. And I was up and running in a few minutes. That's it. -Andrew From nompelis at nobelware.com Thu Oct 31 15:21:13 2019 From: nompelis at nobelware.com (Ioannis Nompelis) Date: Thu, 31 Oct 2019 10:21:13 -0500 Subject: Testing whether I can attach source code In-Reply-To: References: Message-ID: <5048C8F0-FC16-4AFC-B9CF-E558B60216E5@nobelware.com> I was using Elm on the Amiga, HPUX, Sun, SGI Irix, and Linux. I switched to Mutt for IMAP capabilities not found in Elm. So very old school... I configured Mutt several times, using miltiple mail oxes and it was much more trouble than it is worth. The solution was several Mutt instances fired up with their own muttrc over a terminal multiplexer. Do this. Profit. Thank me later. From drewbenson at netjack.com Thu Oct 31 15:38:00 2019 From: drewbenson at netjack.com (Andrew Benson) Date: Thu, 31 Oct 2019 10:38:00 -0500 Subject: Testing whether I can attach source code In-Reply-To: <5048C8F0-FC16-4AFC-B9CF-E558B60216E5@nobelware.com> References: <5048C8F0-FC16-4AFC-B9CF-E558B60216E5@nobelware.com> Message-ID: Elm 2.3 PL11 for a long while, mostly on Sun. Then 2.4. Mutt for about 18 seconds, but then mostly Outlook 10 years, and now Mac/iOS mail. > On Oct 31, 2019, at 10:21 AM, Ioannis Nompelis wrote: > ?I was using Elm on the Amiga, HPUX, Sun, SGI Irix, and Linux. I switched to Mutt for IMAP capabilities not found in Elm. So very old school... I configured Mutt several times, using miltiple mail oxes and it was much more trouble than it is worth. The solution was several Mutt instances fired up with their own muttrc over a terminal multiplexer. Do this. Profit. Thank me later.