Add an Actuator Button to Grafana Dashboard

I think to have found a suitable grafana plugin that can activate/deacttivate an actuator by posting a value with a call api rest. The name of this plugin that can be easily donwloaded and istalled in the grafana is cloudspout button panel plugin. and can be found at grafana.com/grafana/plugins/cloudspout-button-panel


If I understand your question correctly then yes, you can create a windows VM in azure and install jenkins on it the same way as on a local machine.

They're might be some differences in the configuration of the environment for example about security, user access, routing etc


Jenkins deployment key is not associated with the repository

First make sure that you added the public key to the right repo with the right permission, usually I add it to 'Access keys' so the user has read only permission, then to make sure that is working add the private key to you system so you can use it with using git:

ssh-add private-key
git clone <your git repo>

If it's working, add those credential to you jenkins and use them later


PTC with Jenkins on Deployment based on the Build request which raised on PTC

You can pass your data using environment variables or through job parameters.

Here's an example of setting environment variables in a pipeline:

pipeline {
    agent any
    environment {
        FILE = "Makefile"
        SERVER = "example.com"
        VERSION = "v1.1.0"
    }
    stages {
        stage ('build') {
            steps {
                echo "VERSION is: $VERSION"
                bat "script.exe ${version} ${server} ${file}"
            }
        }
    }
}

If you want to use parameters instead you could do so in the following pipeline example:

pipeline {
  agent any
  parameters {
    string(description: 'file', name: 'file')
    string(description: 'server', name: 'server')
    string(description: 'version', name: 'version')
  }
  stages {
    stage("foo") {
      steps {
        echo "VERSION is: ${params.version}"
        bat "script.exe ${params.version} ${params.server} ${params.file}"
      }
    }
  }
}

See the following documentation for building a parameterized pipeline:
https://github.com/jenkinsci/pipeline-model-definition-plugin/wiki/Parametrized-pipelines

If you're using a freestyle job it's slightly different but you can find detailed instructions here:
https://wiki.jenkins.io/display/JENKINS/Parameterized+Build


Rollup status from task to story to feature in Azure Devops?

You may try TFS Aggregator (Web Service). If you or your team have experience with C#, you may try the console project that I`ve developed: Automation of state changing.


Eventually I just got rid of the group_vars all together.
I specified a new role that includes several other roles with 'include_role:' and added the vars directly to those roles with inline-vault variables:

- name: include ipaclient
  include_role:
    name: ipaclient
  vars:
    my_user: user
    my_password: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          123456123456123456123456123456132456...


How to run WSGI Python server without Apache or NGINX

This question doesn't really make sense - WSGI is the mechanism that a web server such as Apache would use to invoke an application or script, such as something written in Python. If you have no web server, then what would be using WSGI in the first place?

Is the question really, how do I take a pre-existing WSGI application and turn it into a standalone? Flask is one such framework that can run as WSGI or using its own internal web server.

But, taking a step back, I still think this is the wrong question. There is no need to blindly apply the "one process per container philosophy" - which is really just some random person's opinion, at the end of the day - if it doesn't make sense in your use case. Think about it for a second. A container containing one process is just... a process. We already had those. Containers work best for bundling things together that logically belong together and administering them as a single "thing". What's inside them should be completely opaque to who- or whatever is managing them. You will never see developers jumping through hoops to follow a "one shared object per executable philosophy" because that doesn't make sense either!

If you have something that already works well with Apache/Nginx fronting your application, then containerize it that way, it will be fine.


ansible AWX is connecting through ssh as root instead of user defined in credential

According to https://docs.ansible.com/ansible/latest/user_guide/playbooks_intro.html

it should be remote_user: if it still does not work, run the playbook with -vvvv and add the log to the Q&A.


I dealt with the same issue. Some people recommended using the Gitlab group "secrets" and using before_script. Since I knew I was also going to deploy and would need other tools on my runner like "helm", I made my own docker container. I still keep my config for my clusters in a base64 encoded group secret but I set it like this in the dockerfile.

ENV kube_config=$kube_config
RUN echo -n ${kube_config} | base64 -d > ~/.kube/config

Now just set an environment variable for your gitlab runner to point to your new image.

KUBERNETES_IMAGE: registry.gitlab.com/MY_USERNAME/MY_REPO_NAME/MY_IMAGE_NAME:latest

Don't forget to test your new container!

stages:
  - build
  - test
  - release

variables:
  CONTAINER_TEST_IMAGE: $CI_REGISTRY_IMAGE/gitlab_runner:$CI_COMMIT_REF_NAME
  CONTAINER_RELEASE_IMAGE: $CI_REGISTRY_IMAGE/gitlab_runner:latest

before_script:
  - apt-get update && apt-get install docker.io -y
  - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY

Build Image:
  stage: build
  script:
    - docker build --build-arg kube_config=${kube_config} -t ${CONTAINER_TEST_IMAGE} .
    - docker push ${CONTAINER_TEST_IMAGE}
  except:
    - master

Test Kubectl:
  stage: test
  script:
    - docker pull ${CONTAINER_TEST_IMAGE}
    - docker run --rm ${CONTAINER_TEST_IMAGE} kubectl get deployments -n kube-system
  except:
    - master

Test Helm:
  stage: test
  script:
    - docker pull ${CONTAINER_TEST_IMAGE}
    - docker run --rm ${CONTAINER_TEST_IMAGE} helm ls
  except:
    - master

Test Docker:
  stage: test
  script:
    - docker pull ${CONTAINER_TEST_IMAGE}
    - docker run --rm -v /var/run/docker.sock:/var/run/docker.sock ${CONTAINER_TEST_IMAGE} docker images
  except:
    - master

release-image:
  stage: release
  script:
    - docker pull $CONTAINER_TEST_IMAGE
    - docker tag $CONTAINER_TEST_IMAGE $CONTAINER_RELEASE_IMAGE
    - docker push $CONTAINER_RELEASE_IMAGE
  only:
    - master

Edit: Missed this part.. I'm using a shell runner.

So post of what I said is probably not relevant. How ever I do think there is some advantages to building and deploying with a gitlab runner directly on a cluster.


How are ansible playbooks supposed to be imported into a collection?

this feature was recently merged to ansible devel branch. It would be posible using:
- import_playbook: testns.testcoll.default_collection_playbook


How to set an existing IAM role to a new instance while spinning in terraform?

you can add role like below

role = "${aws_iam_role.ec2_role_name.name}"

Please check - https://www.terraform.io/docs/providers/aws/r/iam_instance_profile.html


What are some reasons for using an Ansible role for Certbot vs. Ansible's native `acme_certificate` module?

Thanks for linking your question. Here's the github thread for reference: https://github.com/geerlingguy/ansible-role-certbot/issues/87

There are many different ways to get certs from a CA. certbot (what this repo uses) is just one of the ways which uses letsencrypt as a certificate authority. acme_certificate is more generic and if you can't use letsencrypt then it might be a good tool to check out for http-01, dns-01 and tls-alpn-01 challenges.

Certbot is a great way to manage certs from letsencrypt, so if you're needs are fairly standard this is a good choice. It's a higher level package and will also handle auto renewal (assuming you enable it).

it is possible to do renewals with acme_certificate. You could do something similar with letsencrypt https://docs.ansible.com/ansible/2.5/modules/letsencrypt_module.html . certbot just handles a lot of this for you, specifically for letsencrypt.

Personally I think using certbot to help manage certificates is simpler, but that's up to you. certbot is an application that handles the verification process for with the certificate authority, which is very handy. With acme_certificate you'll essentially be doing that yourself. It also automatically can set up automatic renewals which is very nice.


How can I build & maintain a personal, cross-platform 'toolkit' of commonly used applications?

For your own personal use, I think a GitHub repository with all of your bootstrap and installation scripts is a great idea. It serves as a portable way to download and distribute any scripts you need where firewalls and restrictions aren't an issue.

As for within your organization, one suggestion that your IT might be open to as a compromise is to create a machine (barebones, VM, etc.) that is solely dedicated to downloading and storing the artifacts that you need. The criteria for this machine would be:

It has access to download artifacts (files, apps, etc.) from a source (preferably either the open internet or from white-listed addresses).
It is able to store the artifacts in a central location.
It is not able to access internal infrastructure
It is has auditing to track any artifacts that it downloads/stores

With this setup, you could download something like the latest version of Python onto the server (via logging in with credentials, API, etc.) and have it store Python in a central location for your other servers to download and install. You could store your artifacts in a multitude of ways (artifact repository, cloud storage, file server, etc.), but only that server should be able to upload to your storage. Your IT team could even set alerts when a new artifact is downloaded on this server or uploaded to your storage so that they can inspect it.

I think this approach demonstrates a few things to your IT department:

These artifacts are important enough to you and your teams that you are willing to go out of your way to establish a safer and more traceable method to retrieve them.
You have given your IT department an automated way to track who and when an artifact was downloaded in the event an artifact is malicious and needs to be deleted and traced to a source (something they probably didn't have in place before).
The method you propose not only gives you more accessibility to the tools you need to be productive, but you are proposing ways to help your IT team be more productive by not having to manage all aspects of downloading artifacts.


You can do via docker, since it's a test.
Here is an example repository, both docker and vagrant.
https://github.com/jonashackt/gitlab-ci-stack
https://blog.codecentric.de/en/2018/05/gitlab-ci-pipeline/


Minikube - Is there a way to run Windows Containers?

It is not officially supported (as of Minikube 0.25.0, Kubernetes 1.9, January 2018). But there is Beta support for Windows server containers in Kubernetes.

These articles contain more information:

http://blog.kubernetes.io/2018/01/kubernetes-v19-beta-windows-support.html
https://kubernetes.io/docs/getting-started-guides/windows


ansible error 'first argument must be string or compiled pattern'

The task which fails is this one:

 - name: replace hostname in config
   replace:
     path: /opt/agentd.conf
     regexp: #\s+Hostname\=
     replace: Hostname={{hname}}
     backup: yes

Try like this:

 - name: replace hostname in config
   replace:
     path: /opt/agentd.conf
     regexp: '#\s+Hostname\=.*$'
     replace: Hostname={{hname}}
     backup: yes

Note that if you don't add .*$, the task will only replace # Hostname= and you will end up with a Hostname=oldhostnamenewhostname.


While you can use the Release Description field the moment you Start a Release

which fills in the $(Release.ReleaseDescription) variable, I discourage this approach.

The whole point of a Release Pipeline is to automate and having users inputting data is against this approach.
If you do not trust VSTS for Secrets, you can store them in Azure KeyVault or similar and pull them down while releasing. It is not a chance that there are Tasks to facilitate this.


Application monitoring in multiple EC2 instances

It looks like your problem is two fold:

You want to monitor the status of critical applications provisioned on virtual machines
You want to be able to install these checks rather intuitively.

I'll answer these in the reverse order.

The easiest answer for #2 is that you can use configuration management like Chef, Ansible, or Puppet to provision machines. That way you could write a cookbook that as a software package is detected on the system a simple agent based check is dropped in as well. Codifying these things makes them much more manageable in the long term.

In terms of #1 you're probably looking for an agent based monitoring tool. Some examples of this would be Prometheus or Sensu but there's an endless list of them out there. These allow you to describe checks on the server and the master just collects whatever information is given to it.


Puppet master certificate generation not working

First, try setting dns_alt_names in /etc/puppet/puppet.conf:

[main]
    dns_alt_names = www.puppetmaster.com
[master]
    autosign = true

Then see if your puppet-agent -t run works properly. be sure that you have properly set your server on the puppet client in /etc/puppet/puppet.conf too:

[agent]
    server = www.puppetmaster.com


View all AWS S3 buckets and list each buckets storage used

Resolution 1

So I solved this with the following script. I originally posted the question just in case there was an easier way that I was not aware of.

#!/bin/bash
aws_profile=('profile1' 'profile2' 'profile3');

#loop AWS profiles
for i in "${aws_profile[@]}"; do
  echo "${i}"
  buckets=($(aws --profile "${i}" --region your_region s3 ls s3:// --recursive | awk '{print $3}'))

  #loop S3 buckets
  for j in "${buckets[@]}"; do
  echo "${j}"
  aws --profile "${i}" --region your_region s3 ls s3://"${j}" --recursive --human-readable --summarize | awk END'{print}'
  done

done

Resolution 2

Using Dashboards in CloudWatch in the AWS console.

You can then simply specify all S3 buckets and add the numbers stats to show the storage size metrics.

This won't cost you plenty of API calls and can be significantly faster depending on the size of the s3 buckets(takes quite awhile to get the size on very large buckets).

Verdict

Creating the Dashboard (Resolution 2) on each AWS account was the most efficient option for me cause it is way quicker for me to log in and grab the metrics manually from each AWS account than to wait for the scripts API calls to finish. :(


Jenkins does not seem to be running inside a container

If you simply google "jenkins does not seem to be running inside a container", you will see that it's only an informational message. Surely, it could be worded better, but it's pretty much harmless.


Backup Jenkins log files to external cloud storage

Historically, I approached this a bit differently. I added a post-build step to each job, and had it copy the output to S3 at execution time via the S3 plugin.  This blog post covers the moving parts n a pretty granular detail.

You may want to reference

  wget -O console-output.log ${BUILD_URL}consoleText

as the output for the job in question in your build task.


Which container will be targeted when replicas larger than 1

Yes Felix. Loosely speaking it "load balance" across your 5 web container instances. If you are using the ECS / Compose integration and you are bringing up this stack with a compose up this is roughly what happens:

it creates an ECS cluster
it creates task def for the web task and the nginx task
it creates a service for web (with 5 tasks in it)
it creates a service for nginx (with 1 task in it)
it exposes the nginx service via an ALB to the public (because you expose the port 80)
it creates an internal mechanism based on ECS Service Discovery to be able to resolve web inside the cluster (that's how the nginx is able to reach an entity called web).

ECS SD leverages CloudMap/Route53 which will round-robin across the 5 entries (web tasks). You can read more here.


Will data contained withing Persistent Volume survive cluster deletion?

It depends on what your persistentVolumeReclaimPolicy is. If you manually created/defined the PersistentVolume, then the default action is to keep the data. (This sounds like what you're doing.)

If you are using something like EKS on Amazon, your PV's are dynamically generated EBS volumes. In that case, they will be deleted by default.

A little bit more about reclamation: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reclaiming


What is a suitable strategy for provisioning Chef Server through code?

'Suitable' can honestly be whatever works for you. I personally feel its perfectly reasonable to do a manual install of Chef server. It is the CM tool to help provision the rest of your fleet. You will not be setting up Chef server over and over again. You do it once, and then never again, in theory.

That theory breaks down however, in respect to server outages. Eventually, one day, your Chef server will go down. If you are running Chef in HA mode, or you have a very good backup system that allows you to restore the entire server, then you are fine. I don't know what it is for Azure, but with Amazon, you can take nightly snapshots of the entire server and simply restore from the AMI if the server ever goes "hard down".

Let's pretend that you don't run Chef in HA, or you aren't very confident in your backup system, or you just want to tinker... my recommendation would be to leverage Terraform w/provisioner scripts, or Packer with Terraform.

You can get the basics of Chef server setup with some simple commands that only need to be run once. This is perfect for Terraform and it's provisioner scripts. Obviously, it runs the provisioner scripts only once (during server creation) and that's all you need for Chef. If something happens to your Chef server, Terraform can rebuild it from scratch. Although, you'll have to re-key your local client and tie all your instances to the new Chef server. This is a huge headache and I would not recommend ever doing that if you can avoid it. :P


How to see when an AWS IAM role was last used?

Amazon CloudTrail tracks all API use

In CloudTrail create a new trail

You can configure each trail to send log events to CloudWatch: Edit the trail and choose to send Logs to CloudWatch. It will offer you to create a Role for this

https://docs.aws.amazon.com/awscloudtrail/latest/userguide/monitor-cloudtrail-log-files-with-cloudwatch-logs.html

Then, run some calls on your role and wait 5 minutes.

In Cloudwatch, go to your log trail and search for "AssumedRole". Your events will be shown here

There is also documentation on searching Cloudwatch logs with filters through the api https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SearchDataFilterPattern.html


Ansible - using facts from one host to perform actions on another host

I found a solution - the trick here is delegate_to and delegate_facts:
- hosts: all
  remote_user: ansible
  become: false

  tasks:
  - name: read join line from controlPlane
    shell: kubeadm token create --print-join-command
    when: inventory_hostname in groups['workerNodes']
    register: joinCmd
    delegate_to: "{{ item }}"
    delegate_facts: true
    with_items: "{{ groups['controlPlane'] }}"

  - name: join all worker nodes to cluster
    command: "{{ joinCmd.results[0].stdout }}"
    become: yes
    when: inventory_hostname in groups['workerNodes']

The first task kind of reads backwards to my mind - when says 'run this task on all the  workerNode hosts...' but delegate_to says '..but actually run it on a different host'. It works for me though (I've only got 1 host in controlPlane group, not sure of outcome with more hosts).


GitLab runner not able to fetch AWS ECR images in the current AWS account

I use this script in gitlab-ci.yml

image: docker:19.03
services:
- docker:19.03.0-dind
before_script:
- $(aws ecr get-login --no-include-email --region eu-central-1)

... and provide AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY variables into Settings/CI-CD section.
Ensure your region with images is the same where are you logging in.


Google Cloud Storage vs Redis / ElasticSearch

I don't know much about Redis/ElasticSearch, but GCS is not really a database-like solution, it is closer to a file storage solution.

If you're looking for database-like storage Google Cloud offers:

Cloud SQL
Cloud Bigtable
Cloud Datastore
Cloud Spanner

If you expect a modest app traffic or alternating high/low traffic it might be more cost-effective (and simpler) to have a Google App Engine app instead of a GCE-based solution.


What is recommended to document an IT technology stack, including their relationship to one another, in a graph database?

Considering your first paragraph, the organization you're describing is a highly siloed org, which is exactly what a DevOps organization tend to avoid.

  Considering this challenge, within DevOps there is a requirement to
  define and document all the components that constitute the various
  technology stacks within an IT environment. Traditionally this might
  have been accomplished with a propriety CMDB solution.

What you're describing here could be ITIL, which is a management system requiring documentation and you mix it with the fact a DevOps team will usually define the underlying layers as code, as such it gets back to any development documentation with the caveats of Code is Documentation often seen in Scrum methodology for an agile methodology of development (quick and short iterations aiming at minimal working solution at end of the iteration)

Disclaimer for the rest of this answer: I know more of chef and inspec and that's why I take them as exemple here; but they are not the only tools existing on the market, I won't open a debate on them as the better one is the one you're more comfortable with.

As such the rest of the question is a little biased and I, personally, didn't encounter an organization documenting the layer relation you describe more than the infrastructure as code and configuration management system code documentation. (Again, this doesn't mean no-one does it, I just never heard of it).
To illustrate from my company in a chef environment, an application cookbook will declare its dependencies (tomcat, jboss, nginx/php and on which OS, needed mount points for some shared datas and its DB schema name mostly) and expose its services URIs to be consumed by chef for other applications configuration, this sound like defining your 'Finance System' and the documentation for it is on this application cookbook README, with some more files if really needed.

Configuration management systems usually have a central reporting place, "chef-server" for datas an "manage UI" for presentation in chef world "ansible tower" for ansible world to name two of them, but they usually aim more at giving an oversight of the overall managed system than graphing the dependencies.

That said, for chef, the chef-server also act as a CMDB you can query with various tools (it return JSON datas from HTTP requests), the inter applications dependencies can be expressed in various way and there's no 'out of the box' method, each company will have its own way to declare them in the system for configuration purposes and as such you may leverage this to build your graph, but that's on your side.

In an infrastructure as code point of view, the infrastructure needs would be kept with the application, it is still the application who know what it needs under as middle-ware, which OS, with which locale, what are other services dependencies and what services this application offer).

Last thing I can think of if you want to manage those dependencies for documentation only are tools like glpi which is mainly a CMDB and a ticketing system, it takes advantage of documenting assets and their relation to be able to tell what is impacted when you open a ticket saying an application is down. coupled with ng-inventory it allows to query system states and as such can fulfill your query for patch needs, but in my opinion this is an audit system task, like could do inspect integrated within a chef run for exemple, as the next phase would be to fix the outdated systems by updating/patching them.


How to order physical backup of S3 data?

AWS Import/Export is a service that accelerates transferring data into and out of AWS using physical storage appliances, bypassing the Internet. AWS Import/Export Disk was originally the only service offered by AWS for data transfer by mail. Disk supports transfers data directly onto and off of storage devices you own using the Amazon high-speed internal network.

Import/Export was released on May 2009, and described in this blog post - https://aws.amazon.com/blogs/aws/send-us-that-data/

Released on October of 2016 there is also Amazon Snowball, a service for customers who want to transport terabytes or petabytes of data to and from AWS, or who want to access the storage and compute power of the AWS cloud locally and cost effectively in places where connecting to the internet may not be an option.

Blog post describing the snowball service -
https://aws.amazon.com/blogs/aws/aws-importexport-snowball-transfer-1-petabyte-per-week-using-amazon-owned-storage-appliances/

The documentation explains how to prepare a device for export, and how to create the Amazon S3 Export process.

Here are some older docs:

http://docs.aws.amazon.com/AWSImportExport/latest/DG/PackingS3Jobs.html
http://docs.aws.amazon.com/AWSImportExport/latest/DG/createanexportjob.html


Best way to launch 100 identical Docker machines on AWS?

Look at AWS Fargate - it lets you run your Docker containers without having to spin up the EC2 instances for an ECS cluster. You simply schedule the Tasks (or Services), each gets is own IP address and they can talk to each other.

You can also use AWS Service Discovery to facilitate the registration and lookup of the tasks' IP addresses.

BTW Note that the default Fargate concurrency limit is 20 Tasks/Services - if you need 100 of them you'll have to raise a Service Limit increase support ticket.

We've got it raised to 200 concurrent tasks because our processing runs 160+ containers in parallel, all on Fargate without a single EC2 to manage.

Update - how to spin up 100 containers

I can tell you how we do it with Fargate - in our CI/CD pipeline we build the container image and upload to ECR. Part of the source is also a CloudFormation (CFN) template and in the Deployment stage of the pipeline we create/update CFN stack which creates ECS TaskDefinition with all the container parameters.

Then we've got a scheduler job written in Python and triggered by CloudWatch Event (cron-like scheduler) that looks up the TaskDefinition ARN and runs 160+ Tasks from that single TaskDef. You can give each Task an extra parameter, or override some config, etc. Or not and just spin it up.

We used to use EC2/ECS and had to do an extra step first - scale out the ECS AutoScaling group to tens of nodes to support our workload and then at the end scale it back in to 0. Other than that it's the same whether you run it on Fargate or on EC2/ECS.

Hope that helps :)


How to introduce(join) new worker node to an existing k8s cluster with selective scheduling only

Possibly you are looking for this  feature which is closed with no action since there are ways to work around this problem.

  kubeadm already allows to set taints for the joining node using the configuration file at join time (check my other answer for details on how to do that)

One Other way to achieve this is by running join as usual but taking few extra steps as below after join

  1) After successful join , Force Drain the new node to make sure it removes unwanted pod which might have scaled to it post join operation kubectl drain <node name> --force

  2) Drain command will auto cordon off the node. (Note : also consider special handling to remove any daemonset which might have scaled to this node after join)

  3) Now Taint the new node as needed and then uncordon it

  4) Add nodeselector plus tolerance to pods you want to be scheduled on this new node.

Logs of above steps for reference are listed below.

On below example i have joined a new node-02 and assume that some pod were scaled in an existing deployment so new node got used immediately on join, so before adding we need drain action and then taint the node so going forward no scheduling can be done unless pod has tolerance to the taint added

Nodes Before Drain or Taint

$ kubectl get all -o wide
NAME                         READY   STATUS    RESTARTS   AGE   IP               NODE                NOMINATED NODE   READINESS GATES
pod/nginx-86c57db685-8cdkx   1/1     Running   0          27s   192.168.58.120   k8s-node02-calico   <none>           <none>
pod/nginx-86c57db685-xnh6q   1/1     Running   0          65s   192.168.182.78   k8s-node01-calico   <none>           <none>

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   2d    <none>

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES   SELECTOR
deployment.apps/nginx   2/2     2            2           65s   nginx        nginx    app=nginx

NAME                               DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES   SELECTOR
replicaset.apps/nginx-86c57db685   2         2         2       65s   nginx        nginx    app=nginx,pod-template-hash=86c57db685

Drain The node

$ kubectl drain k8s-node02-calico
node/k8s-node02-calico cordoned
evicting pod "nginx-86c57db685-8cdkx"
pod/nginx-86c57db685-8cdkx evicted
node/k8s-node02-calico evicted

$ kubectl get nodes
NAME                STATUS                     ROLES    AGE   VERSION
k8s-master-calico   Ready                      master   2d    v1.17.2
k8s-node01-calico   Ready                      <none>   2d    v1.17.2
k8s-node02-calico   Ready,SchedulingDisabled   <none>   2d    v1.17.2

Taint the new node

$ kubectl taint node k8s-node02-calico newnode=nobodyallowed:NoSchedule
node/k8s-node02-calico tainted

Now we are good to uncordon the node, no more pods should be able to use this noe unless they add toleration to the new taint.

$ kubectl uncordon k8s-node02-calico
node/k8s-node02-calico uncordoned

$ kubectl get nodes
NAME                STATUS   ROLES    AGE   VERSION
k8s-master-calico   Ready    master   2d    v1.17.2
k8s-node01-calico   Ready    <none>   2d    v1.17.2
k8s-node02-calico   Ready    <none>   2d    v1.17.2

We test by scaling the deployment and nothign should be added on node02

$ kubectl scale deployment --replicas=3 nginx
deployment.apps/nginx scaled

$ kubectl get all -o wide
NAME                         READY   STATUS    RESTARTS   AGE     IP               NODE                NOMINATED NODE   READINESS GATES
pod/nginx-86c57db685-vm8ql   1/1     Running   0          3m30s   192.168.182.75   k8s-node01-calico   <none>           <none>
pod/nginx-86c57db685-xnh6q   1/1     Running   0          10m     192.168.182.78   k8s-node01-calico   <none>           <none>
pod/nginx-86c57db685-zh2sj   1/1     Running   0          8m19s   192.168.182.76   k8s-node01-calico   <none>           <none>

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   2d    <none>

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES   SELECTOR
deployment.apps/nginx   3/3     3            3           10m   nginx        nginx    app=nginx

NAME                               DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES   SELECTOR
replicaset.apps/nginx-86c57db685   3         3         3       10m   nginx        nginx    app=nginx,pod-template-hash=86c57db685


How to export EC2 snapshot into S3 bucket?

EBS Snapshots aren't really disk image files.  They're logical containers of pointers to chunks of compressed and deduplicated backup data that EBS stores in S3 when you create snapshots -- but these are stored in buckets that EBS owns and controls.

They are user-accessible via only one mechanism -- creating a new EBS volume from the snapshot.  When you do this, the new volume is linked back to its source snapshot and EBS starts loading the data onto the new volume in the background, from the snapshot storage in S3.  If you access a part of the volume that EBS hasn't yet restored, it immediately goes and fetches the chunks and puts them on the volume.

To create a disk image file of a snapshot, you create a new volume from the snapshot, attach it to a running instance without mounting it, and then use the same tools you would use for a standard hard drive -- tools like dd to read raw blocks from the snapshot, and write them to a file, which you can then download from the instance, e.g. using scp.


Auto build and upload to AWS new docker image after commit in repo

how can I automate this process? I have an script to push dockerimage to AWS through ecr.

You are on the right track. Yes, this should be an automated process from Git push or when someone merge a Pull Request.
An important aspect of this is that this "automated process" should be on a dedicated system - such that it is repeatable in a consistent way. Typically this is called a "Continuous Integration Server" or just "Build Server".
Some popular choices for this kind of process:

Jenkins - old but popular
Tekton - Kubernetes native - but still in development
GitHub Actions - probably most interesting if your Git repo is on GitHub
Bitbucket Pipelines - probably most interesting if your Git repo is Bitbucket
AWS CodeBuild - limited but may be interesting if you use AWS
GitLab CI - probably most interesting if your Git repo is on GitLab

What you need to do is to configure your Git repository system (e.g. GitHub, Bitbucket, Gitlab or AWS CodeCommit) to initiate Pipelines on the Build system (a product of those listed above)


`UnauthorizedResourceAccessException` when calling `aws workdocs create-user`

While the WorkDocs API describes create-user as creating a user in the Directory Service (and does), it also activates the user as a WorkDocs user.  As a result, it can only be called when the directory is associated with WorkDocs (i.e. when the directory is "enabled" in WorkDocs).  These errors can occur when WorkDocs is not enabled.
These errors can arise, for example, if you're trying to create users for WorkSpaces using the WorkDocs API since a comparable API isn't available for WorkSpaces.  To enable WorkDocs, do one of the following:

Use EnableWorkDocs when registering the Directory with WorkSpaces
Manually create a site on the WorkDocs page

WARNING:  Even if you allocate no storage, it appears these users will incur WorkDocs charges unless you also deactivate_user.
WARNING:  EnableWorkDocs is not fully reversible from the API i.e. you cannot delete the Directory Service from the API if a WorkDocs site exists and the site cannot be removed from the API (likely for data protection reasons)


Is there a limitation in AWS to open an IPSec VPN connection?

For me it looks to be tunnel misconfigured -  "there is a message about protocol mismatch"

Regarding the limitations quick google took me here: [strongSwan] strongswan limits


Jenkins transferring file using publish over SSH plugin but file not show in remote development server

Please try with double // before var in remoteDirectory: '/var/www/sockets'.


What tools do you use to monitor applications and business-specific data? (apart from Jenkins)

We use Datadog, with metrics pushed from important operational processes using the statsd protocol ( http://docs.datadoghq.com/guides/metrics/ ), so we aren't too deeply tied to Datadog on the application side.

In Datadog's interface we can then configure monitors to alert on any number of things, including anomaly detection if defining thresholds is impossible.

So for instance, we have many tasks that run on a hourly basis, they'll post a datapoint to a given metric (app.task.task_name), and we will then monitor that this metric is consistently posting at one data point per hour. If the rate changes, we get alerted.


How can I simulate failure of an AWS Availability Zone?

This (slightly old in cloud years) question on Hacker News provides a few suggestions:

Remove the security groups from all instances in one zone, which should technically make them all appear 'down'
Adjust the VPC route tables to black hole the subnets in one availability zone
Disable an availability zone from the load balancer (docs)

You could also probably adjust the Network ACL attached to the relevant subnets to deny all traffic.

I'm not aware of an existing, available tool that does any of this automatically, though.


Chef Provisioning for canary deployment and orchestration

You should take a look at BOSH.  Its the tool that is used by CloudFoundry, its services and a distro of Kubernetes called Kubo for installation,  management and "Day 2" operations.

It's basically a declarative, cloud-agnostic orchestration tool that features rolling updates, canary deployments, scaling, monitoring and self healing.  It can monitor VMs as well as processes on those VMs (i.e. services) and will make sure that the deployment is running as you specified it to in the deployment manifest.

In order to do all of this (especially the monitoring and self-healing bits) it has a client-server architecture which is deployed with a cut-down version of BOSH itself called bosh bootloader or bbl for short.  You use this to deploy the BOSH director.  You talk to the bosh director by installing the bosh-cli (brew install bosh-cli on a mac).

For you deployment you would first need to create what's called a BOSH release and this can, admittedly, be a little daunting if you are not familiar with BOSH but as CF, its services and Kubo are all open source there are tons of references out there.  There are also lots of pre-backed releases and stemcells (OSes).

After creating your release you upload it to your bosh director and bosh deploy.  To upgrade it you upload the next version of your release and bosh deploy.  To patch a security vulnerability you upload the latest blessed stemcell from bosh.io and bosh deploy.  I am sure you get the picture.

If you want to go the next level then there is a good getting started guide here.


Environment variable names, when my development environment has multiple projects

check out https://direnv.net/ it sets environment variables based on the directory you are in.

other options are:
1.docker-compose
2.plain bash to set the variables.
3. bash source .env
4. use prefixes but you would still need to set the environment you are in via an environment variable, so back to options 1-3.

if you are going to store passwords or keys in the env variables locally that are sensitive i.e not a localhost db pass, use a secrets manager like https://pkhub.io/usecases/environments, or you could script up something with aws' secrets manager


How do I install local application dependencies in a Kubernetes manifest?

You can't add run-time dependencies on an already build image using a yaml manifest.
You might attach/exec to an already running pod and install them, but it makes no sence, since you need to make sure that when the pod is re-created you do it again. The much better and right way is simply to describe it in your Dockerfile.

Kubernetes is not created for such kind of hacky moves. It simpy maintains state that you describe.  Essentially you need to add an instruction that adds git in your Jenkins dockerfile, build your new image and refer to it inside the manifest.

Commands and args are used if you want override the image's default entrypoint and its arguments.


Issue with custom domain SSL certification

Can you obtain a secure certificate for someone else's domain or subdomain without their help - hopefully not.

One of the core security pillars of a secure certificate is that it ensures that you are actually connected (through an encrypted connection) to the the site you wanted to connect to. For this purpose the minimum check every certificate issuer makes is to check for ownership of the domain by the entity requesting it.

For "manual" issuers, this often involves uploading a file to a web server on this domain, creating a DNS record or clicking on a link in an email to the registrant, administrator or technical contact in the whois record of the domain.

You can replicate this process for your needs for your SaaS application by using LetsEncrypt: It allows you to either

upload a file to a folder on a web server for the domain: In this case you would advise the client to point an A record (or probably better CNAME) to your server and once resolved, you can issue the certificate with letsencrypt by copying the file yourself
create a specific DNS record, which again allows you to create the certificate.

Lets encrypt certificates are valid for 90 days, but as you will need the A record in place in anycase to run your SaaS, you will be able to renew the certificate yourself regularly.

P.S.: Not too long ago, subsidiaries of Symantec were found (twice) to not always have checked domain ownership and then did not appear to take remedial action consistently (twice). This led to the "distrusting" of all Symantec certificates in Chrome and Firefox after giving owners of these certs some time to create new certificates.


Increase concurrent users by limiting container resources with docker-swarm

In general: No you can't. At least not on the Docker level.
You can maybe code your app in a way that it monitors such metrics and adjusts itself but Docker isn't able to.
What Docker CAN do is limit the app to 50% and kill it if it uses more than that by utilizing resource limits:
docs.docker.com - containers/resource_constraints
docs.docker.com - docker-compose resource limits


Is JS an reasonable choice for a DevOps internal tool (i.e. cloud-formation runner, scripts)?

Definitely. Especially when using AWS services, the aws-sdk for JavaScript is superb.

Writing code using JavaScript in a shop that has both frontend people writing JS in some framework (Angular/React/Vue/...) and backend people using Node.js removes barriers. Just pushing to use JS provides the org. with even more developers who don't have barriers to reading and fixing infrastructure code.

For example, I am quite proficient in using Python, Ruby, Shell, JavaScript (even Perl). But when it comes to fixing or adding something to HashiCorp's tools like Terraform, I just don't have the time to learn Go sufficiently enough - so I don't do it. If a tool like Terraform was written using JavaScript, I would have added my contribution for that missing service a long time ago.

JavaScript support is excellent in AWS Lambda as well, and that is quite a big
deal. For example, one of the ways to make CloudFormation do things you never thought it could is to write an AWS::CloudFormation::CustomResource implemented in a Lambda function.

Another good example is streaming CloudWatch Logs into ElasticSearch. If you ever try to press that button you discover that AWS just creates a JavaScript function for you in Lambda.

JavaScript is everywhere in DevOps and AWS, doing the same thing with Java, or C# or even Go would take much more time.

Should you bring a "DevOps Engineer" in the future (please don't), and that person does not know JavaScript while all your code is the backend and frontend is in JavaScript then simply don't hire such a person. It is a bad fit for both you and them.


It is.

You can install the ZIP installation and unzip it in the EFS.
Alternatively, install using Debian/RPM and mount the EFS to /var/opt/jfrog/artifactory in advance.

Alternative approach can be to only configure the Filestore to use the EFS.


Can I check whether ansible is on last batch of rolling update (serial: 25%)?

To solve this, I printed vars and ansible_facts, and eventually stumbled on the ansible_play_batch variable, which is a list of hostnames in the current batch. ansible_play_hosts_all gives the full list of hostnames. (Do not use ansible_play_hosts for this purpose, because as of Ansible 2.7, it works but the docs say it gives something completely different.)

The question: Is this the final batch?
Rephrasing based on the data: Is this batch == the last part of the host list?
Or to make it simpler: Is the end of this batch == the end of the host list?

In Ansible, the check for that is:

tasks:
  - pause:
      seconds: "{{ health_check_healthy_threshold * health_check_interval }}"
      prompt: Waiting to allow AWS health checks to pass
    when: ansible_play_batch[-1] != ansible_play_hosts_all[-1]

  - debug: msg="This is the last batch."
    when: ansible_play_batch[-1] == ansible_play_hosts_all[-1]

Side note: I used the following playbook for testing, to simulate having many/few hosts without bothering to set them up. I couldn't see how to specify the hosts as a list in the playbook, so I added them to a group, then executed the group:

---
- hosts: localhost
  connection: local
  tasks:
    - add_host: { name: host1, groups: test_hosts }
    - add_host: { name: host2, groups: test_hosts }
- hosts: test_hosts
  connection: local
  serial: "25%"
  tasks:
    - debug: var=ansible_play_batch


How to deal with untested features in the develop branch before deployment?

You could use the same tools used in Trunk Based Development (TBD) to carry work-in-progress in the trunk (the only non-release TBD branch with a longer lifespan):

feature-toggles/feature-flags
branch by abstraction:

  "Branch by Abstraction" is a technique [1] for making a
  large-scale change to a software system in gradual way that allows you
  to release the system regularly while the change is still in-progress.

These would allow merging a feature in develop/master at any time, without the feature being actually (completed/functional and) enabled.

If technically these are implemented using configurable means then at any time the software can be tested both with and without a feature enabled, simply by flipping the config flag in the respective test environment. Otherwise special builds may be needed to enable them (still a lot easier than doing a branch merge with the feature's full/partial code).

And you could then run all your feature testing in develop/master to get your features enabled, one at a time.

But if you go in this direction - why would you even struggle with git_flow and run into all these problems? Just go TBD and CI/CD all the way :)


Deploying VMs using roles and hosts in Ansible based on operating system

So I managed to keep the same structure roughly, and managed to separate how the machines are generated, however still is a bit fuzzy and can probably be improved to be more efficient!

site.yml

- name: Apply common configuration to all nodes
  hosts: all
  roles:
    - common

- name: Configure and deploy test node 1
  hosts: build-test-node
  roles:
    - build-machine-test-1

- name: Configure and deploy test node 2
  hosts: build-test-node-2
  roles:
    - build-machine-test-2

environments/test/groups

[win_test_1_nodes]
win_build_machine_1

[mac_test_1_nodes]
mac_build_machine_1

[win_test_2_nodes]
win_build_machine_2

[mac_test_2_nodes]
mac_build_machine_2

environments/test/hosts

win_build_machine_1   ansible_host=......
win_build_machine_2   ansible_host=......
mac_build_machine_1   ansible_host=......
mac_build_machine_2   ansible_host=......

environments/test/meta

[win_test_1_nodes]
[mac_test_1_nodes]
[win_test_2_nodes]
[mac_test_2_nodes]

[win:children]
win_test_1_nodes
win_test_2_nodes

[mac:children]
mac_test_1_nodes
mac_test_2_nodes

[build-machine-test-1:children]
win_test_1_nodes
mac_test_1_nodes

[build-machine-test-2:children]
win_test_2_nodes
mac_test_2_nodes

This solution allows more test nodes to be generated with separate roles added to them, and can just be added to the correct group for the machine to be provisioned.


Are distributed distributions like NixOS able to replace docker in terms of autoscaling?

Autoscaling has nothing to do with NixOS or Docker. This is just the feature of cloud backed applications.

However it is true that the best results are achievable using immutable infrastructure, but you do not need neither of these tools to achieve such. Instead you can build your own AMI (using Packer) and then use AWS CodeDeploy to deploy your application on each of the instances.

These tools are orthogonal, so you can use them, but there is no problem with using autoscaling without them.


Have you checked the following link: https://wiki.jenkins.io/display/JENKINS/Step+by+step+guide+to+set+up+master+and+agent+machines+on+Windows
I advise you to configure a node on the master ( manage --> manage nodes ) and set the launch method to  launch agent by connecting it to the master. After that you have three methods to connect the Windows agent to Jenkins master, the easiest way will be to lanch it from a browser.

This should work enven with jdk 11.


Jenkins Testing PR's - allowing broken builds

On GitHub - you can't. Basically, their PRs are temporary branches, and there's no check if base commit is HEAD of branch to merge in.
This problem is solved with gating systems like Zuul, which puts all PRs into queue to run tests.

You are talking about systems for managing and storing code, while the issue is in CI scope (i.e. Gitlab flow has this issue described in "Testing before merging" paragraph), Zuul is made exactly to fix this problem, because the amount of merges to OpenStack codebase per day is large (year ago they had 6 zuul instances to run tests for merges).


How to know which user created the GCE Instance

Note: I'm not a GCE user yet, the answer is based solely on documentation.

You could be Viewing Audit Logs in the Google Cloud Console, more specifically the Admin Activity logs:

  Admin Activity logs contain log entries for API calls or other
  administrative actions that modify the configuration or metadata of
  resources. For example, the logs record when VM instances and App
  Engine applications are created and when permissions are changed. To
  view the logs, you must have the IAM roles Logging/Logs Viewer or
  Project/Viewer.

You would be looking for the User identities in audit logs:

  Audit logs record the identity of the user performing logged actions.
  The identity is held in the AuthenticationInfo field of AuditLog
  objects.

Ideally you'd be looking for the GCE instance creation events, but some/all of those may be too old for the logs retention policy. Then maybe search for VM instance start or other relevant events.

You could also check and maybe cross-reference the audit logs info with that from Usage Reports or exported billing info.

If this is a regular, on-going activity you may want to setup an automated log exporting and processing pipeline.


How to automate the magento2 installation process too via docker

Instead of creating a dockerfile yourself you could also pull a docker image, e.g. alexcheng/magento2.


How does using packer and terraform in a CD pipeline compare to docker images built from git?

You're comparing apples with oranges according to your description. A strict comparison could be packer toward dockerfile, and/or terraform toward kubernetes yaml for example.

Main difference between your docker pipeline and your packer/terraform one is the architecture behind them.
What is putting the burden here is that your AWS infrastructure doesn't take advantage of auto-scaling groups rolling updates, or your 'stage-gates' in go does disrupt them.

You're not comparing docker with terraform, you're comparing rancher with your AWS infrastructure choices. If you don't use the equivalent bricks of rancher load balancing and deployment groups (kubernetes wording here) which are Elastic Load Balancers and Auto Scaling Groups in AWS you can't compare them.
That's saying a car and an bicycle does the same thing (transportation) with a better result for the car, put a motor on the bicycle and you'll have a far different appreciation and something less biased.

Your observation is pointing out that your usage of AWS facilities is sub optimal, terraform (or whatever provisionning system) can't help you more than running things, if your architecture is not made (in the terraform file) to allow rolling updates from this facility, no automatic system will fix it for you as you'll have to code it again (test instances status, etc.).

Of course rancher pre-bake most of those, making them invisible to you. This also mean you are enforced in rancher methodology (well, swarm, kubernetes or messos architecture to be exact).
That's the main difference, on one side you have an infrastructure already ready to serve with rancher and you can't run outside it, and on the other side you have to think your architecture before hand. If you were to run docker containers on a bare docker host (without any orchestrator on top) you'll end up with the same architecture design to think before hand and the same caveats you're seeing on AWS.

AWS is not strict on how you use it, it's an infrastructure as a service with no opinion on how you use it, whereas rancher enforce a bunch of services around your own container to save you the design time as long as you accept the constraint within it.

Hope this help seeing the problem from an upper level and as such I think the answer to your questions is that you were missing a pattern ;)

For the records: we're doing deployments on AWS with cloudformation (native version of terraform, with its own caveats) from  a push on a gitlab server and it works nicely, the work to get proper cloudformation templates and the script to run it was non negligible. This to say the tools are not the problem, it's all a design choice at a point, same results can be achieved, arncher comes with its own caveats and design choices to make when you integrate it into an existing infrastructure.


How to avoid branchageddon with large organisations?

As Michael mentioned, offer a standard solution based on release versions/numbers, with a reasonably long lifespan for your industry (maybe interleaved with one or more shorter lifespan intermediate versions, if it makes sense for your typical customers).

Give your customers the option to embark on this standard release track, maybe with a decent migration deadline in place.

If they insist on a full-custom branch support strategy just charge them accordingly, to properly cover all your extra costs for offering such full-custom support - it only makes business sense. Some customers will migrate to reduce their costs (which will help you reduce the number of custom branches), some won't.

Variable support billing, growing progressively with the age of the release versions from which the custom branches originate can also be an incentive for customers to migrate to newer branches faster, helping with faster closing of the older custom branches. This can help reduce the number of custom branches per customer - if you have customers that simultaneouly run multiple versions of your software.

Make sure you don't fall into the trap of doing full branch merges from/to any of the release branches (both standard and custom), all changes to them should be either individually developed or cherry-picked merged fixes.

Because each of these branches will gradually diverge from each-other, the number of hotfixes requiring customisation/individual development will grow exponentially (plain cherry-pick merging will fail). You need to take the development cost for these into account.

With no (significant) branch merges in the picture you can (and should, I cannot stress its importance enough) build fully automated CI/CD pipelines for these branches, accompanied with a good hotfix tracking/management system in place, making the hotfix delivery just routine (or almost).


Auto-managed docker-compose volume not shared among containers

Define the volume in the following way

volumes:
  - ${local_path}:/data

or using docker volumes

version: '3.7'

volumes:
  db-data:

services:
  one:
    image: alpine
    command: sh -c 'echo one >> /data/db.txt && cat /data/db.txt | wc -l'
    volumes:
      - db-data:/data


Why is IF failing with satl state file from Jenkins?

The jinja syntax error is caused by the {{ around your job var.
Those curly braces are not required as you are already inside jinja block, noted by the {%.

So to get your code working just replace  your {{ job }} by job.
This should be working.


How does 'Environment Management' work in a DevOps world?

I've never heard of an "Environment Manager."  Release Management, on the other hand, has historically been consolidated to a single person or team.

In a DevOps model, Release Management is more of a process supported by elements from both Dev and Ops.  Organizations at high levels of "DevOps maturity" don't really have a distinct Release Management event- the act of releasing is baked into the Continuous Delivery pipeline (see "push on green" What is "Push on Green"?)

It's possible the OP is also referring to Configuration Management, where environment-specific configurations are managed manually, or by tools like Puppet or Chef.  That work is traditionally an Ops-managed task, with higher-maturity orgs handling that through code.  Responsibility may be shared between Dev and Ops, or there may be a dedicated SRE team that deals with Config Management.


How to prevent failure from introduced changes?

This is pretty deep in to "opinion based" but here goes:

While most definitions of "DevOps" are independent of specific tools and techniques, it is very often combined with things like test automation (e.g. Jenkins, Travis-CI) and deployment automation (e.g. Chef, Capistrano, Kubernetes). Those two together act as a forcing function to push up quality levels of software by making it both easier to use tests to ensure things don't break and making it easier to fix breakages when they do occur.

That said, I really doubt that number captures the full story. Like so many things, "software quality" is easy to talk about but difficult to define.


az aks browse doesn't open browser automatically on WSL and only showing JSON instead of the dashboard

I would expect it to work like az login and open up a new tab in default browser. But somehow it doesn't work like that. I think the fix must come from azure CLI.

Temporarily, For az aks browse you can setup an environment variable on WSL.

export BROWSER="/mnt/c/Program Files (x86)/Google/Chrome/Application/chrome.exe"

In this case next when I run az aks browse it will open up the dashboard in new chrome tab.

You can use this variable to use your favorite browser. I would also add this variable to .bashrc or profile to be added on every session.


Is there an equivilent of a Kubernetes ingress for Docker Compose?

Kubernetes ingress is implemented by changing the configuration of another container or the cloud load balancer to route traffic to the service port/ip. Docker by itself is fairly lightweight, only runs the containers you start, and does not have external dependencies on things like your cloud load balancer. The solutions to setup a reverse proxy inside a container are almost identical to the kubernetes nginx based ingress controllers, with the key difference being the configuration of that reverse proxy.

There are a variety of docker based reverse proxies that configure themselves, including docker-flow-proxy, nginx-proxy, and Docker-EE ships with interlock that configures HA proxy. Probably the easies solution I've seen to implement is traefik which can be configured to monitor the docker socket for changes and automatically reconfigure based on labels set on your containers.

One important difference from Kubernetes is that docker networks are not flat, you will need to explicitly place the proxy container on the same docker network as your application in order for them to be able to route traffic between each other.


Why use EXPOSE in Dockerfile -- since you can bind to all ports anyways

Docker's EXPOSE documentation addresses this specific point:

  The EXPOSE instruction does not actually publish the port. It
  functions as a type of documentation between the person who builds the
  image and the person who runs the container, about which ports are
  intended to be published. To actually publish the port when running
  the container, use the -p flag on docker run to publish and map
  one or more ports, or the -P flag to publish all exposed ports and
  map them to high-order ports.

Pay attention to the last sentence, if you expose multiple ports then -P becomes useful to avoid setting multiple -p on the command line.


Azure Dev Ops delete releases not going to production

The interface is a little vague, but if I go to Pipeline > Edit then click "Retention" at the top, I can set values there that work.
Note: The updates/clean up only happen at... some time???  I'm not 100% sure when, but builds aren't immediately cleaned up if the policy changes.
The Days to Retain a Release box and the Minimum Releases to Keep influence each other, and the system seems to keep the build if either meets the criteria.
So, I set the UAT stage to 1 day / 1 release, so it always keeps AT LEAST 1 build, and always keeps all builds for AT LEAST 1 day.
I set the Prod stage to 3 day / 3 release, so it always keeps AT LEAST 3 builds, and always keeps all builds for AT LEAST 3 days.
Now it's much cleaner:

I hope this helps someone in the future! :)


DNS Issues with Cloud formation template

Which flavor of Linux are you using? (eg Ubuntu/Debian, RHEL/CentOS etc)
Also which version? (eg Ubuntu started using the "netplan" configuration by default in 20.04)
At first glance, it looks like your DNS nameservers are hosed somehow.
And can you post the results for doing a: netstat -nutpl?
Look at which PID/Program Name is listening on port 53 (systemd-resolve? dnsmasq?) and check its nameserver configuration.
Maybe check here: sudo cat /etc/resolv.conf  and make sure the nameserver(s) listed there are what you think you should have them.
or check the DNS-related services are even running with something like:
Ubuntu:  sudo service systemd-resolved status (NOTE BOTH of the 'd' endings)
CentOS/RHEL: sudo service dnsmasq status
(or whatever is using port 53, which is the default DNS port on most OSes)


I tried to change docker image and added couple steps for git tag. Below is the answer worked for me,

image: maven

stages:
- build
- deploy
- tag

maven_build:
stage: build

  script:
   - mvn clean package
   artifacts:
  paths:
   - target/*.jar

  after_script:
   - ls -a
   - cd target && ls -a
   - git --version
   - git remote remove origin
   - git remote set-url origin https://"username:passwd"@gitlab.com/accountname/projectname
   - git tag -a 1.0.15 -m "Version created by gitlab-ci Build"
   - git push origin 1.0.15
  only:
    - master


Azure DevOps - Conditionally Build and Deploy

You can create build pipeline triggers to trigger your builds based on branch along with path triggers. In your release pipeline, you can set triggers and filters to decide which artifacts to be deployed.


AWS has something built in called cloudwatch alarms that can handle the basics.
You could also use a Lambda Function. Give it access to the ASG, and then have cloudwatch configured so once the min amount of traffic starts coming in it sends an Event to the Lambda.
{
  "ASG": "MyASGName"
  "Action": "Start"
}

And once the traffic goes below the min.
{
  "ASG": "MyASGName"
  "Action": "Stop"
}

You could also build on this over time to control Vertical and Horizontal scaling.
EDIT: I actually prefer the Lambda. I always seem to run into these weird use cases where we can't shut down any of the instances, we have to do some math or investigation on which instances to shut down. In these situations a python lambda running boto3 is really helpful.


Puppet directory sync does not delete absent files

According to https://tickets.puppetlabs.com/browse/PUP-4759 the issue seems to be solved.


You can try below options to debug this further.

First verify values.yaml and check the values for serviceAccount (default value is true ) that means the helm chart when deployed will create a service account for the deployment. When you create a pod, if you do not specify a service account, it is automatically assigned the default service account in the given namespace.

serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # Annotations to add to the service account
  annotations: {}
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name:

Try to deploy your helm chart with create: false and verify if it solves your issue.

Else remove the file hello-world/templates/serviceaccount.yaml and mark serviceaccount as create: false in value.yaml and pass name as default service account of your deployment try to deploy as below

Note I have removed the serviceaccount.yaml

~/hello$ tree
.
└── hello-world
    ├── charts
    ├── Chart.yaml
    ├── templates
    │   ├── deployment.yaml
    │   ├── _helpers.tpl
    │   ├── ingress.yaml
    │   ├── NOTES.txt
    │   ├── service.yaml
    │   └── tests
    │       └── test-connection.yaml
    └── values.yaml

4 directories, 8 files

set create field as false under values.yaml

$ cat hello-world/values.yaml | grep -i serviceaccount -A 10

serviceAccount:
  # Specifies whether a service account should be created
  create: false
  # Annotations to add to the service account
  annotations: {}
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name: default

You can fetch your list of serviceaccounts available as below

$ kubectl get serviceaccounts
NAME      SECRETS   AGE
default   1         2d4h

Once deployed helm list will show the namespace used to deploy your chart and serviceaccount from the name space is used for deploeyemnt

$ helm list
NAME                    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART               APP VERSION
hello-world-1580665905  default         1               2020-02-02 17:51:45.631330487 +0000 UTC deployed        hello-world-0.1.0   1.16.0

Get chart details and values post deploeyemtn as below

$ helm show chart hello-world
apiVersion: v2
appVersion: 1.16.0
description: A Helm chart for Kubernetes
name: hello-world
type: application
version: 0.1.0

$ helm show values hello-world | grep -i serviceaccount -A 10
serviceAccount:
  # Specifies whether a service account should be created
  create: false
  # Annotations to add to the service account
  annotations: {}
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name: default

podSecurityContext: {}
  # fsGroup: 2000

Get deployed resources as below

$ kubectl get all -o wide
NAME                                          READY   STATUS    RESTARTS   AGE    IP               NODE         NOMINATED NODE   READINESS GATES
pod/hello-world-1580665905-84b9d4d469-xv5mx   1/1     Running   0          118s   192.168.58.215   k8s-node02   <none>           <none>

NAME                             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE    SELECTOR
service/hello-world-1580665905   ClusterIP   10.108.73.21   <none>        80/TCP    118s   app.kubernetes.io/instance=hello-world-1580665905,app.kubernetes.io/name=hello-world
service/kubernetes               ClusterIP   10.96.0.1      <none>        443/TCP   2d4h   <none>

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE    CONTAINERS    IMAGES         SELECTOR
deployment.apps/hello-world-1580665905   1/1     1            1           118s   hello-world   nginx:1.16.0   app.kubernetes.io/instance=hello-world-1580665905,app.kubernetes.io/name=hello-world

NAME                                                DESIRED   CURRENT   READY   AGE    CONTAINERS    IMAGES         SELECTOR
replicaset.apps/hello-world-1580665905-84b9d4d469   1         1         1       118s   hello-world   nginx:1.16.0   app.kubernetes.io/instance=hello-world-1580665905,app.kubernetes.io/name=hello-world,pod-template-hash=84b9d4d469


Jenkins pipeline running using containers on an orchestration platform

You can run your application on a particular node of the cluster by defining it in the application level. For this, you need to add the "placement " parameter in the yaml file and specify the node's name to it, then it will deploy the container on that node.

Secondly, what type of scheduling have you implemented on Swarm CLuster? Let me clear your thought, there three of them: Spread, Bin-pack, and Random. By default, it is Spread, which identifies a number of containers on each node and deploys the new container on the node that has a small number of containers. Check the rest yourself.

Both of these points will solve your problem.


What about using a CI tool? Create a parallel job and the jobs will run parallel. For example, one could use Jenkins:

https://jenkins.io/blog/2017/09/25/declarative-1/

stage('run-parallel-branches') {
  steps {
    parallel(
      a: {
        echo "This is branch a"
      },
      b: {
        echo "This is branch b"
      }
    )
  }
}

but there are other CI tools that could do the same.


Is it possible to create a module that either creates a resource or sources a data source, based on a boolean value?

You can use the count meta-parameter to achieve the effects of an if-else statement.  It is talked about in detail in the link below and I also pulled one relevant example out.

I'm not 100% sure if you can give both the data source and resource the same name without a conflict; but I assume it would work.  It may be dependent on the resource you're creating though, not sure.

https://blog.gruntwork.io/terraform-tips-tricks-loops-if-statements-and-gotchas-f739bbae55f9

Excerpt:

# This is just pseudo code. It won't actually work in Terraform.
if var.give_neo_cloudwatch_full_access {
  resource "aws_iam_user_policy_attachment" "neo_cloudwatch_full" {
    user       = aws_iam_user.example[0].name
    policy_arn = aws_iam_policy.cloudwatch_full_access.arn
  }
} else {
  resource "aws_iam_user_policy_attachment" "neo_cloudwatch_read" {
    user       = aws_iam_user.example[0].name
    policy_arn = aws_iam_policy.cloudwatch_read_only.arn
  }
}

To do this in Terraform, you can use the count parameter and a conditional expression on each of the resources:

resource "aws_iam_user_policy_attachment" "neo_cloudwatch_full" {
  count = var.give_neo_cloudwatch_full_access ? 1 : 0
  user       = aws_iam_user.example[0].name
  policy_arn = aws_iam_policy.cloudwatch_full_access.arn
}
resource "aws_iam_user_policy_attachment" "neo_cloudwatch_read" {
  count = var.give_neo_cloudwatch_full_access ? 0 : 1
  user       = aws_iam_user.example[0].name
  policy_arn = aws_iam_policy.cloudwatch_read_only.arn
}


Is there a way to acquire metadata about an Amazon RDS database instance using only SQL?

I don't have a SQL Server on AWS to check, but you could see if it's in xp_msver, or figure it out from ProcessorCount, ProcessorType and PhysicalMemory in that sproc.


Accessing External REST api From Wthin Kubernetes Cluster

A service in kubernetes is (mostly) used for internal communication. If you want to access an external resource by domain/ip you can just do so. If you have an IP address and you would like to have some sort of domain connected to it, you could you am service with the external name field set to the up address (we use that for database purposes). An endpoint is automatically created for you when you create an service so you don’t nees to bother with that


docker compose & azure key vault - passing secrets to a container on startup

Found this -->  Passing secrets to a Docker container

Environment variables appear the preferred method - will need to run a prebuild script to fetch the secrets and then follow this

https://docs.docker.com/compose/environment-variables/

@Mods feel free to close as a duplicate


The blog post you linked to in your comment which describes using YAML to generate Pipelines is about an unofficial 3rd-party Jenkins plugin which adds this feature; neither native Jenkins nor official plugins support using YAML to generate Pipelines.

pipelines or declarative pipeline (do other type of pipelines exist in jenkins?) are defined using a syntax that is based on an extension of groovy

There are two types of natively-supported Pipelines: Scripted Pipelines (which are pure Groovy scripts) and Declarative Pipelines (which use a Groovy-based DSL).


How does continuous integration relate to continuous delivery / deployment?

Continuous delivery and continuous deployment both take continuous integration one step further, by adding a 'deployment to production' step to the process.
The difference between continuous delivery and deployment is that for delivery this step is done manually and for deployment is it automatic.

Difference between Continuous Integration, Continuous Delivery and Continuous Deployment. Picture copied from codeproject.com

Whether you do continuous delivery or continuous deployment is very much an implementation choice. If you do continuous deployment, changes in code will be deployed automatically after the acceptance tests are passed. This may or may not be desirable for your product. With continuous delivery, people can make a choice whether a particular code change  is deployed or not (and possibly where exactly it is deployed).

Since the difference between continuous delivery and deployment is small and many people are unaware of the exact difference, the two terms are sometimes used interchangeably.


Found an easy solution using the index function:

tags = { Name = "Company0${index(var.vpc_cidrs, each.value) + 1}" }


Amazon EKS: using spot instances for worker nodes

It can be done by modifying the Cloudformation Template (which is mentioned in this document) in the LaunchConfiguration section to specify a spot price.

NodeLaunchConfig:
  Type: AWS::AutoScaling::LaunchConfiguration
  Properties:
    SpotPrice: "20" # <=== Here
    AssociatePublicIpAddress: 'true'
    IamInstanceProfile: !Ref NodeInstanceProfile
    ImageId: !Ref NodeImageId
    InstanceType: !Ref NodeInstanceType
    KeyName: !Ref KeyName
    SecurityGroups:
    - !Ref NodeSecurityGroup
    BlockDeviceMappings:
      - DeviceName: /dev/xvda
        Ebs:
          VolumeSize: !Ref NodeVolumeSize
          VolumeType: gp2
          DeleteOnTermination: true
    UserData:
      Fn::Base64:
        !Sub |
          #!/bin/bash
          set -o xtrace
          /etc/eks/bootstrap.sh ${ClusterName} ${BootstrapArguments}
          /opt/aws/bin/cfn-signal --exit-code $? \
                   --stack  ${AWS::StackName} \
                   --resource NodeGroup  \
                   --region ${AWS::Region}


This is a great question as some of the terminology can be very confusing - particularly surrounding the (unfortunate) use of "SFTP" to describe multiple similar technologies.

Will the real SFTP please stand up

This comes mainly from the fact, that RFC 913 defines one SFTP protocol, the "Simple File Transfer Protocol" which offers no encryption and went largely unadopted, with users favoring TFTP and FTP depending on their needed application.

Since the FTP protocol, defined in RFC 1738 in 1985 is completely encrypted and packet sniffers are able to dumping plaintext passwords from this protocol, Some thought that it would be a good idea to secure this protocol by implementing FTP over an SSL connection. In 1997 a mechanism for doing that was released in RFC 2228 which often was referred to as SFTP.

Meanwhile, back at the ranch...

Meanwhile, a similar security issue was occurring around the telnet protocol - passwords and sensitive information were unencrypted and subject to sniffing. In response, SSH-1 was released in 1995 which provided an encrypted equivelent to telnet for use over network connections.

This protocol became very popular very quickly and before a formal specification allowing for FTP over SSH was released, users of the SSH protocol had implemented a version of BSDs rcp (remote copy) over SSH. This protocol was extremely limited however, which led to a revision in SSH - SSH2 which added an SSH File Transfer Protocol (not to be confused with FTP over SSH, another SFTP) because, according to wikipedia:

  Compared to the SCP protocol, which only allows file transfers, the SFTP protocol allows for a range of operations on remote files which make it more like a remote file system protocol. An SFTP client's extra capabilities include resuming interrupted transfers, directory listings, and remote file removal.

And In fact, the IETF set up a working group to codify a formal standard for SSH-2 (released as RFC 4251) but

  Eventually, development stalled as some committee members began to view SFTP as a file system protocol, not just a file access or file transfer protocol, which places it beyond the purview of the working group.

And eventually SSHFS was released as just that.

Conclusion

So the answer is a bit complicated.

SSH and SFTP and SCP could be regarded as completely different protocols depending on which SFTP you are referring to and which version of the SSH protocol you are referring to. Yet, the command for actually doing a file transfer remained "scp" even after the SFTP and SSH-2 protocol was developed. If one is referring to SSH-2 however, these could arguably all be considered the same protocol.

Side note: SSH was released by Tatu Ylönen of the Helsinki University of Technology in 1995. As Linus Torvalds released Linux in 1991 as a student at the University of Helsinki until 1996, it was kismet that SSH would be so popular and widely used in Linux.


Git Remote Access in Azure Pipelines, possible?

I've had a similar issue with git commands in Azure Pipelines.  The reason is that any commands that access the origin server, such as git push need to done by a user that has the credentials to do so.  In your situation, your PowerShell script is running in the user context of Project Collection Build Service (yourProjectName) and it does not have the permissions to contribute to your repo by default.

See this answer, I think it will fix you up: https://stackoverflow.com/a/56542631/11767897


Docker - Mount a volume from a container to an other (equivalent volumes_from) in docker-compose 3

The Problem: Named volumes are initialized on container creation when the volume is empty, and from the first container that starts up using that volume. That initialization includes all files, directories, and file metadata like owners and permissions. If there is content in the named volume, docker will not initialize it again since that would defeat the purpose of persistence.

Option A: When you have multiple containers sharing a volume to share updated build time content like you have, my first suggestion is to modify the build to place the relevant data in each image. Depending on what is being shared and between what images, you may be able to use shared image layers to avoid ever needing to copy the data.

Option B: When placing the data in each image and avoiding the shared volume is infeasible, then you likely want a process to copy data from the image into the volume as part of your container entrypoint. This has two advantages. First it works when the containers startup in either order, so you can eliminate dependencies between container startup order (though you may find it necessary to include some wait step in the other container entrypoints if they fail when run before the data is copied). And second, your copy can update the volume when existing data is already there.

I've implemented a load-volume and save-volume pair of scripts in my docker base image that you can reuse in your own images to implement this second option. I typically ran into this issue when using tmpfs volumes to support read-only containers, but the scripts have been made generic enough to support your use case. Note that your scenario likely involves deleting existing data from the volume, and I've implemented that using rsync for performance, so that needs to be installed in your image to work.


Without seeing the contents of your two modules I'm guessing a bit, but it looks like you have an AWS VPC declared in your vpc module and some subnets declared in your prod_subnets module and you are asking how the configuration of the subnets can get access to the VPC ID.

If so, the answer is that the vpc module must export the VPC ID as an output value and then the prod_subnets module must accept the VPC ID as an input variable.

In your vpc module, you can declare a vpc_id output value like this, for example in a file modules/vpc/outputs.tf:

output "vpc_id" {
  value = aws_vpc.production_vpc
}

In your prod_subnets module you can declare a vpc_id input variable, for example in a file modules/vpc/modules/subnets/production/outputs.tf:

variable "vpc_id" {
  type = string
}

Then in your existing modules/vpc/modules/subnets/production/production.tf file, on line 162, change the vpc_id argument for the subnet to refer to that variable:

  vpc_id = var.vpc_id

Finally, you must then edit the top-level file whose source code you shared in your question to pass the value between the two modules, like this:

module "vpc" {
  source = "./modules/vpc"
}

module "prod_subnets" {
  source = "./modules/vpc/modules/subnets/production"

  vpc_id = module.vpc.vpc_id
}

module.vpc.vpc_id means to take the value of the vpc_id output value from the vpc module.


How frequently do PaaS:VPS agencies upgrade OS_release in comparison to PaaS:shared-webserver?

From my experience this is very provider specific and requires evaluating the SLA offered for a particular service.

For example the siteground cloud server offerings shown at
https://www.siteground.com/cloud-hosting.htm state: We preinstall and maintain all the software you need on your server.

While offerings from liquidweb they make it clear they will customize the support level to fit your requirements (the custom levels of management are detailed here). You can chose the support level for the product you purchase, whether it's a cloud VPS or dedicated server hardware.

IMHO most cloud products will probably be vendor managed and provide you a way to perform needed customization. But general statements are not useful when it comes to spending money. You have to confirm what your vendor is providing.


How to securely pass sensitive data to EC2 Spot instances?

When you're running ephemeral workloads like you would on a spot instance the best bet, if you don't want to bake an ami, is going to be a secret store of some kind.

The two most common methods I've seen for doing this are:

If your provisioning tool (terraform, cloudformation, salt cloud) allows it then you can create a kms key and use it to encrypt secrets in user data. Once the instance has started it can decrypt them using that key.
You can use a secret store like you mentioned. You said that AWS Secrets Manager would get pricey so you're probably going to want to use Vault. It's not that hard to set up and can use IAM permissions to allow your instances access to secrets. Using it also opens other avenues to doing secret storage right.


How to prevent hug of death on EC2 Instance?

If you want something quick to get this sorted without much more knowledge I'd recommend elastic beanstalk. It's another AWS application which will handle the load balancer configuration and instance scaling for you.

There's no extra cost on top of the load balancer and instances so you can keep using t2 type instances but let elastic beanstalk scale as much as you want to help.

Auto scaling isn't instant and in times of spiking traffic will take a short amount of time, minutes normally, to be able to handle a spike but it will be much better than manually scaling the instance size and is really easy to get to grips with.


Trigger PagerDuty alert after a number of incidents

The easiest way to build this would be to have some sort of metrics-based alerting system like Prometheus, datadog, etc.  Those systems allow you to have a counter that gets incremented and you can see how many incidents happened in a given period of time on a pretty graph.  Most metrics systems will tie nicely to PagerDuty for the alerting piece.  Having any of these metrics systems functioning will also give you a historic baseline for your systems that is often incredibly handy in figuring our what went wrong and when.

The threshold should probably be number of incidents over a given period of time.  If you just pick an arbitrary number, say 100, for this you will always eventually get to that limit.  If you see 10 incidents in an hour that could be bad for you.

The hardest part of this would be that I don't think PagerDuty makes it easy to access metrics about the number of incidents.  You can look at pretty graphs in their UI, but there's no easy way to feed that into your metrics system.  You would need to add a bit of code to the other places you have creating incidents and have them increment this counter.


CI/CD ARM Templates - Should the cloud infrastructure and resources be checked on each deployment?

if you are doing ARM then follow the replace ideology if you can manage to have things stateless but if not then maybe go via the Terraform route than ARM IMHO


How to have kubernetes (re)build containers on deployment similar to openshifts ImageStream

You need something that will do basically CI/CD. There are few options (in no particular order):

Gitlab + autodeploy (or custom deployment)
skaffold + git-sync + kaniko

There are more, but basically, you need a component to watch for changes, something that knows how to build an image, and something else that can deploy it (some of these might be done by the same tools)

Helm will only do the deployment, but nothing else


Assign bug from different repo when creating a pull request

There are actually many ways to link work items to other objects in Azure DevOps.  Microsoft's documentation is pretty extensive on this topic.  However, for the particular task at hand, linking code commits (I'll assume Git is in use for this example), can be done in several different ways.

In your commit message - By including the work item ID in the commit message prefixed with the hash mark # (i.e. #12345), when the commit is pushed to the server, the work item and commit are linked automatically.  Then, when this commit is part of a pull request that is created, the linkage is made visible as part of the pull request.
Within the pull request during creation, or afterward when edited by the creator, you can add work items directly to the pull request by searching either by ID or by keywords.
Several other methods can be discovered through editing a work item or right in Visual Studio.


Jenkins: Access job/plugin configuration values inside pipeline

This is possible, but you'll have to use scripted Pipelines, not declarative, disable the Groovy sandbox for pipelines, and dig deep into the Jenkins API.
For instance, I wrote a Pipeline job that edits the parameters of a second job which has a full name of "SCA/deploy_to_environment/master".  Here's a snippet of that code - I'm eliding how to create the "Param" objects in the first line for brevity, this snippet is just to give you an idea of what a similar use case looks like:
def paramProp = new ParametersDefinitionProperty(envParam, pgParam, infoParam)

def job = Jenkins.instance.getItemByFullName('SCA/deploy_to_environment/master')

// There is no "setProperty" - we need to remove and replace.
// removeProperty is safe to run even if no matching properties are set on the job.
// However, this method only removes one property per method call.
// Theoretically, this means the job could accumulate multiple conflicting
// ParametersDefinitionProperty properties.
// TODO: run removeProperty in a while loop to eliminate all matching properties for extra safety.
job.removeProperty(ParametersDefinitionProperty)

job.addProperty(paramProp)

So to fetch the values instead of simply editing them, you would need to instead call getProperty (API doc here), passing the class name of the GitLab Project parameter classes. I suspect the class name you'll need is available somewhere in the source code or the API docs for that plugin, but I'm not sure.

However, I have to ask, why do you want to do this?  What's the big picture here?
It sounds kind of like you want to share a single Jenkinsfile or build pipeline among multiple repositories.  I also have this use case at my job - we need to have one Jenkinsfile shared across nearly 200 repositories.  If this is similar to your need, then consider using Jenkins Shared Libraries.  You can put all of your build pipeline code into a single shared library, then call the shared library in a single line from the Jenkinsfile in your individual repositories.


How to run Dockerized graphical applications on a headless server with packages required for display remotely on clients as part of a separate image?

You can't run a program running on x11 without a x11 buffer somewhere, be it vnc, xvfb or whatever. You will always need x11 libraries on the system running the program or it couldn't work at all. That's akin to  trying to run a C program without libc on the system.

Using system libraries from a different container with a volume is a bad idea, as it create a hard dependency on both containers using the same libraries, and is likely to crash somewhere when both systems try to acquire a lock on the library to load it.

I assume with a lot of tweaks it should be doable, but the added complexity is a real footgun and create more dependencies than should exist.


Using docker with users leaves files belonging to the subuid/subgid

docker run -ti --rm \
  -v $PWD:/workdir \
  -v $HOME/.ssh:/root/.ssh \
  --user $(id -u):$(id -g) \
  dockerizedxyz "$@"

As long as you're not doing anything funny with the entrypoint in the container, this will leave files owned as the user that invoked Docker.


Rename build pipeline .yml files in Azure DevOps

As suggested by @030, I have prepared two screenshots that show how to connect the .yml files with the build pipeline.

Rename the .yml file in the repository
Open the pipeline editor

Select the file from the dropdown

Select the in step 1 renamed *.yml file and click on save.

After the existing .yml is assigned, the pipeline works as before the renaming.


How many executors can I have for a Jenkins Windows build node?

You can specify the executors of your node in his configuration.
Manage Jenkins -> Manage Nodes -> Configure Node -> Number of build processors
In environments where we only run scripts etc. we have one executor per core.
But if you have large build jobs and the performance is important for you, you should read this article from the jenkins documentation.
https://www.jenkins.io/doc/book/scaling/architecting-for-scale/#Calculating-how-many-jobs


docker-compose local variable declare and use

https://docs.docker.com/compose/environment-variables/#the-env-file

One could set a variable in an .env file:

TAG=1.2.3

and call it in a docker-compose file

docker-compose.yml

version: "3"
services:
  web:
    image: nginx:${TAG}
    ports:
    - 80:80

docker-compose up

returns:

Pulling web (nginx:1.2.3)...
ERROR: manifest for nginx:1.2.3 not found

No need to run docker-compose to check the config

According to the documentation one could run:

docker-compose config

to verify the docker-compose config

networks: {}
services:
  web:
    image: nginx:1.2.3
    ports:
    - 80:80/tcp
version: '3.0'
volumes: {}


Check connectivity of 5000 + servers from ansible and get list of hosts which are connectable and which are not

Maybe a simple shell script would be more useful. If must to use ansible, it could be something like that (assuming you have all servers in group testall):

- hosts: testall
  connection: local
  gather_facts: false
  tasks:

    - wait_for:
        timeout: 2
        port: 43
        host: '{{ (ansible_ssh_host|default(ansible_host))|default(inventory_hostname) }}'
      register: waitfor
      ignore_errors: true

    - copy:
        content: |
          {% for host in groups['testall'] %}
          {% if hostvars[host].waitfor.failed %}{{ hostvars[host].inventory_hostname }} {% endif %}
          {% endfor %}
        dest: /tmp/bad_file
      delegate_to: localhost

    - copy:
        content: |
          {% for host in groups['testall'] %}
          {% if not hostvars[host].waitfor.failed %}{{ hostvars[host].inventory_hostname }} {% endif %}
          {% endfor %}
        dest: /tmp/good_file
      delegate_to: localhost

hosts that have this port opened will be in /tmp/good_file, others are in /tmp/bad_file


How to check accepted cipher suites by an Azure App Service?

Not sure if this is the right answer but let me give it a try with the limited knowledge i have on this.

these are the resources that I found when I am looking up for tls enabling for my app service. please see if any of this works.
msdn
how to configure ciphersuite for app service
enforce tls vrsion
The client (browser) gives a list of cipher suites it can handle to the server and the server selects one, the decision is passed on to the client during the handshake.
so all the servers won't accept all the cipher suites, it based on server config to choose.


Run scripts which are stored in GitLab from Jenkins

If you are using Gitlab merely as a source code repository, and you are triggering the jobs when there are changes to the repository.

The Jenkins Gitlab plugin documentations states:

  This plugin is a build trigger that allows GitLab to trigger Jenkins builds when code is pushed or a merge request is created. Configuration done on a per-job basis.

So, you can trigger the builds. But to answer your question:

  I put all the commands in Shell to execute. Can I store those commands in GitLab and execute from jenkins?

The answer is yes, but there is some leeway. It depends on how you've configured the jobs. If they are freestyle jobs, have them execute a shell command running the script in your repository. You can do this via the web-interface the Jenkins job editor.

However, I wouldn't recommend this as best practice. Instead, I would recommend having a pipeline description in the repository. See the plugin docs on github for more details on configuration.


Check netstat -tlpn on the VPS, can you see 0.0.0.0:3306 or something like that? If so, check the firewall. Both the firewall in the operating system and the firewall from the VPS provider (if there is any). As a last resort (and not a bad idea) - you can connect to your VPS via VPN - there are good openvpn tutorials on the internet. The reason why the VPN is a good idea - especially for a development server is that you may need to allow the user from everywhere (if you do not have a static IP address) and if you accidentally create some 'testuser' with 'testpassword' anybody from the internet will be able to connect to your db. So use good passwords, limit access to your public IP and use VPN if possible.


What is the cheapest way to get CI for a Qt project?

One of the major advantage of the container technology is standardisation: easily replicating the same environment wherever a specific container is used, be it, for example, on your own machine during development or on the CI machine during the execution of the CI/CD pipeline.

Building and using custom containers is a pretty standard way of including OS-level dependencies (like libraries and toolsets, for example) which aren't included in a service provider's default container images. Depending on the provider the choices for the base image you'd use might be limited.

If you have a CI/CD provider which accepts custom-built containers (not all of them do, at least not yet) and you can meet their custom container restrictions (if any) then building the custom container for your case is IMHO a valuable investment. Not only for the CI execution, but even for developing of your app and ultimately for showcasing the container technology as a development technology itself :)


GitLab - trigger pipeline to build on specific runner

There currently isn't a solution for building on a specific runner in GitLab, but there is an issue open for Sticky Runners, which hopefully will be out in the next 3-6 months according to the Milestones!

The work around I've done so far to build a project on a specific runner is to use the GitLab Runner API, in a rather hacky way, along the lines of:

Get all project runners
As I know I've deployed the latest runner, that would have the highest runner "number"
Pause all the other runners associated with the project in question
Trigger the pipeline to build on the latest runner
Poll the GitLab API to get the status of the pipeline
Once that succeeds, resume all other runners!
If the pipeline fails, remember to resume the paused runners...


It sounds like your goal is to have a variable that takes a list and to validate each of the elements of that list. If that's true, then the approach would be to write an expression that returns true only if all of the elements are valid. For example:

variable "names" {
  type = list(string)

  validation {
    condition     = can([for v in var.names : regex("test$", v)])
    error_message = "All names must end with \"test\"."
  }
}

The above relies on the fact that regex will fail if the given string doesn't match the given pattern, and that the for expression will fail if the given expression fails for any v. can then captures the failure and returns false, making the validation fail.