Tips: How to automate OwnCloud files synchronization on Windows with WinSCP

Posted by on 08/14/2019 No comments

Tips: How to automate OwnCloud files synchronization on Windows with WinSCP 1
I had a task to automate customer files backup from Windows 10 desktop to NextCloud server. The main idea to keep offline backup that will be updated automatically few times a week (not every day). Main requirement for this was to reduce human interaction with the process as much as possible and exclude some files from backup process all together.
Initially I was not concerned about this – after all there is official Nextcloud Windows desktop client along with command line tool, ownCloud desktop client is also compatible and also has command line tool included, so I thought to myself out of these two I should be able to put together some kind of automation.
Well, to my dismay both official clients proved to be completely useless in unattended automation. Whatever webDAV protocol limitation there were, both command line clients had it, for example simple task of synchronizing single directory, say d:\testsync to remote/testsync was impossible because neither client could create 1st level folder on remote server. It could be underlying limitation or bug of Qt implementation of WebDAV protocol, but figuring that out was out of scope of my task.
I needed reliable and compact (this is not the requirement but always welcome, considering that official clients were dragging with them about 100Mb of Qt core libraries and are useless for my purposes) WebDAV client for Windows with automation and I needed it yesterday.
Then, I discovered that WinSCP supports WebDAV and allows scripting advanced enough to help me with my task.
After some testing I cam up with the following system:

  1. CMD script started by Task Scheduler, executes WiSCP console tool with the script, in the loop until it exits with “success” result

  2. WinSCP console util executes script that synchronizes local storage with NextCloud server

    filemask covers exclusions

This solution synchronizes about 200Gb of data from single Windows every 2-3 days. It’s pretty fast, compact and the best part of it is that it transfers only changed files. Case closed.

Rant: Why DevOps is not an answer? Because there was no question

Posted by on 03/06/2019 No comments

Why do I think that DevOps is not going to last too long as it is now widely marketed and hyped? Because there was really not that much demand for it in the first place – it was hyped and brought up by project management and some marketing people that always know better how programming and software development should be done.
I’ve seen some IT hypes rise and go

  • XML is the answer for all your data representation needs (not)
  • Javascript/Ruby/C# is the only programming language you need to know (Python is thriving, C and C++ are alive and kicking)
  • Unix is obsolete – Windows NT can cover all your server demands (yes I am that old)

As the experience shows in 2-3-4-5 years hype dies out and most of things return back to the way they were before, may be with the slight new twist. Of course there is always some market share leftovers staying on so much hyped technologies just because big investments were made into it and these could not be unmade without some unpleasant discussions so at the end it was just decided to call this innovations a success.
What I think, that the whole DevOps deal was an attempt of some management to cut costs ™ namely make software developers do sysadmin tasks without hiring professional sysadmins and basically without salary rise. Round of applause ensued, big bonuses distributed and reports of innovations issued. Nice and shiny face of the capitalism here. New tools and technologies came out shortly after and placing DevOps into the title and somewhere on the resume will ensure recruiters interest in you.
Why do I think this is totally wrong? From my more then 20 years experience I could say that software developers and system administrators have not only different skill set but also very different mindset and way of solving problems. You know what they say – “If you are a hammer most of the problems start looking like a nail”? Software developers/programmers are set to solve a problem with writing new code or modifying an existing code. All side problems like configuring development environment, setting up networking, backups and information security are seen as an obstacles to the final and ultimate goal – software development. So you would naturally assume that performing “Op” tasks will be quick, inaccurate and well yeah mediocre in order to achieve an ability to do “Dev” part (19% of Docker images with empty root password is the nice illustration to issue). You want an example? Here you go – saving AWS credentials in github repository is widely known security problem nowdays, and guess what – in most cases that was done by some CI/CD tool or some high and mighty DevOp that was harmless and respected programmer in his previous life.
I wouldn’t say that DevOps is total pure evil in itself – there are some good ideas in it, but dumping all system administration tasks on developers usually wouldn’t lead to any good outcome. Just like widely popular tape recorders with bundled radio long time ago – both functions were way below average.
I hope that at the end the common sense will prevail.

Tips: How to make Google Chrome to use less RAM

Posted by on 03/04/2019 No comments

Tips: How to make Google Chrome to use less RAM 2
It’s no news – everybody knows that, especially serious web surfers who have many tabs open – sooner or later if you will not restart your browser completely, no matter how much RAM do you have on your computer, Google Chrome will consume it all and start crashing – partially by killing extension processes or completely – by killing/trashing multiple tab processes. Google developers are constantly working on improvements trying to make Chrome use less RAM but so far they have mixed results.
Read more »

Windows: What do I want to see as default browser

Posted by on 03/01/2019 No comments

Windows: What do I want to see as default browser 3

Why Did I bother with it at all?

As I wrote before it is more convenient and secure to have something small, fast and feature limited as your default browser (valid decision for all OSes out there).
Well times go by and nice small Qtweb got outdated with development on it stopped about 6 years ago and new standards (namely SSL/TLS) and new vulnerabilities came out, so I decided
Read more »

Sysadmin: Letsencrypt renewal htaccess redirect bypass

Posted by on 05/27/2018 No comments

With increasing role of HTTPS websites (Google pushing everybody to run only HTTPS websites considering regular HTTP as insecure) the service provided by Let’s encrypt becomes critically important. But there is a catch – once you get the certificate and redirect your site to HTTPS using .htaccess you will get a problem renewing certificate because 301 redirect breaks the challenge verification and the command

gives an error about authorization problem.
Read more »

Sysadmin: Virtualmin and Letsencrypt integration

Posted by on 11/16/2017 No comments

Sysadmin: Virtualmin and Letsencrypt integration 4Sysadmin: Virtualmin and Letsencrypt integration 5
If you are using webmin excellent system for managing virtual host configuration it would make perfect sense to integrate with it the popular certificate authority Letsencrypt that issues completely free SSL certificates.
There are few initial steps that has to be made nside Webmin in order to make it utilize Letsencrypt SSL certificate issuing process for configured virtual hosts. I have successufully configured and used Webmin version 1.831 and certbot-auto 0.12. YMMV.
Read more »

Sysadmin: PHP-FPM modular config

Posted by on 10/06/2017 No comments

Sysadmin: PHP-FPM modular config 6
When you have multilple PHP-FPM pools configured on the server you usually store the configurations under /etc/php-fpm.d. When you have a lot of sites this directory starts looking pretty crowded.
Although when you look inside the typical PHP-FPM pool configuration file you can easily notice that there about only 4 lines that make a difference – everything else is absolutely identical.
Read more »

Sysadmin: How to configure PXE server on VMware workstation

Posted by on 09/20/2017 No comments

Vmware Workstation is the perfect candidate for configuring local PXE server for testing and development – it contains independent DHCP server that is an equivalent to ISC DHCP v2. All you need is to have your own VM that will be serving as TFTP server for network boot images, there are plenty of instruction on how to do that on Linux (.
I decided to configure my Vmware workstation for windows (windows 7 in that case) DHCP as PXE server using vmNAT network adapter, since I already have Linux VM in vmNAT network which I can configure for TFTP server.
The config file is %SYSTEMDRIVE%\Users\All Users\vmnetdhcp.conf, in order to enable PXE you will need to add 4 lines.

  1. 2 Lines at the top right after comments
  2. And another 2 lines inside the subnet block that is related to your VmNAT adapter network

    You will need to assign static IP to your Linux TFTP server so that network booted VMs will be able to access it.
  3. Now you can go to Administrative tools->Services and restart Vmware DHCP server – if everything correct then it will restart without complains.
  4. In order to configure your TFTP server use this, or this instruction

Sysadmin: How to install Letsencrypt on CentOS

Posted by on 09/18/2017 No comments

Sysadmin: How to install Letsencrypt on CentOS 7
Since Letsencrypt started their services life became much easier (or less). You can get free valid and secure SSL certificates that are recognized by majority of browsers, but for that you need to do some initial configuration and keep in mind that if you don’t renew all these nice certificates will turn into pumpkin after 2 month.
Here is the one liner command to install certbot-auto in CentOS 6 (requires additional Python 2.7 from EPEL repo)

Almost, the same for the CentOS 7 just without Python.

Here is how quickly request/retrieve certificate for your website on domain.com that is hosted under /home/domain/public_html from command line (presuming that the site is answering on this server already e.g. DNS and web server configured properly)

If the request was successful your new certificates could be accessible from /etc/letsencrypt/live/domain.com, where
privkey.pem – is the certificate key, cert.pem – bare certificate fullchain.pem – certificate along with the whole CA chain.
Don’t forget that these are valid for 2 month only. You’ll need to run

after that to get valid certificates.

Sysadmin: how to test your web site before switching DNS

Posted by on 04/25/2017 No comments

Sysadmin: how to test your web site before switching DNS 8 How to test your web site after moving it to the new location, before switching DNS and “going live” there? This situation happens every day, even multiple times a day with single page placeholder sites and huge complicated web portals. I thought I’ll just put together simple instruction to refer people to it instead of explaining it over and over every time.
So, in order to test your web site on the new server, before switching DNS settings to the new location you will need to fool your desktop browser to look into new IP location. Here is how to do it.
Read more »